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The  Military  Operations  Research  Society  (MORS) 


The  purpose  of  the  Military  Operations  Research  Society  (MORS)  is  to  enhance  the  quality  and 
effectiveness  of  classified  and  unclassified  military  operations  research.  To  accomplish  this  purpose,  the 
Society  provides  media  for  professional  exchange  and  peer  criticism  among  students,  theoreticians, 
practitioners,  and  users  of  military  operations  research.  ITiese  media  consist  primarily  of  the  traditional 
annual  MORS  Symposia  (classified),  their  published  abstracts  or  proceedings,  special  mini-symposia, 
workshops,  colloquia  and  special  purpose  monographs  and  other  publications.  MORS  publishes  two 
quarterly  periodicals,  PHALANX  and  Military  Operations  Research.  PHALANX  is  the  MORS  bulletin  and 
Military  Operations  Research  is  a  referred  journal.  The  forum  provided  by  these  media  is  directed  to 
display  the  state  of  the  art,  to  encourage  consistent  professional  quality,  to  stimulate  communication  and 
interaction  between  practitioners  and  users,  and  to  foster  the  interest  and  development  of  students  of 
operations  research.  In  performing  its  function,  the  Military  Operations  Research  Society  does  not  make  or 
advocate  official  policy  nor  does  it  attempt  to  influence  the  formulation  of  policy.  Matters  discussed  or 
statements  made  during  the  course  of  its  symposia  or  printed  in  its  publications  represent  the  positions  of 
the  individual  participants  and  authors  and  not  of  the  Society. 

The  Military  Operations  Research  Society  is  operated  by  a  Board  of  Directors  consisting  of  30  members,  28 
of  whom  are  elected  by  vote  of  the  Board  to  serve  a  term  of  four  years.  The  persons  nominated  for  this 
election  are  normally  individuals  who  have  attained  recognition  and  prominence  in  the  field  of  nulitary 
operations  research  and  who  have  demonstrated  an  active  interest  in  its  programs  and  activities.  The 
remaining  two  members  of  the  Board  of  Directors  are  the  Immediate  Past  President  who  serves  by  right  and 
the  Executive  Vice  President  who  serves  as  a  consequence  of  his  position.  A  limited  number  of  Advisory 
Directors  are  appointed  from  time  to  time,  usually  for  a  one-year  term,  to  perform  some  particular  function. 

MORS  is  Sponsored  by: 

•  The  Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

•  The  Director,  Assessment  Division,  Office  of  the  Chief  of  Naval  Operations 

•  The  Director  of  Command  and  Control,  Deputy  Chief  of  Staff,  Air  and  Space  Operations,  US  Air 
Force 

•  The  Commanding  General,  Marine  Corps  Combat  Development  Command 

•  The  Director  of  Force  Structure,  Resource  and  Assessment,  The  Joint  Staff 

•  The  Director  Program  Analysis  and  Evaluation,  Office  Secretary  of  Defense 

SCS  International 

Established  in  1952,  the  Society  for  Computer  Simulation  (SCS)  International  is  a  non-profit  volunteer 
driven  corporation  (Simulation  Councils,  Inc.).  SCS  International  is  the  multidisciplinary  forum  dedicated 
to  research,  development,  and  applications  of  simulation.  Members  are  engineers,  scientists,  managers, 
educators,  and  students  from  many  countries,  with  organized  regional  councils  in  the  United  States  and 
Canada,  the  European  Economic  Community,  Pacific  Rim,  China,  and  Mexico. 

SCS  International  sponsors  several  conferences  each  year  providing  learning,  presentation,  and  networking 
opportunities  at  the  forefront  of  computer-based  simulation  technologies  and  applications.  Most  conference 
proceedings  are  published  as  books. 

The  monthly  journal  SLMULATION  is  a  refereed  publication  filled  with  technical  articles,  industry  news, 
editorials,  an  artificial  intelligence  column,  a  calendar  of  events,  conference  information,  and  an  annual 
directory  of  simulation  vendors.  TRANSACTIONS  OF  THE  SOCIETY  FOR  COMPUTER  SIMULATION  is 
a  highly  refereed,  scholarly  journal  presenting  theoretical  and  practical  applications  of  archival  value. 
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SIMVAL  99  Purpose/Objectives 


The  Simulation  Validation  (SIMVAL)  99 
Woikshop  was  held  at  The  Johns  Hopkins 
University  Applied  Physics  Laboratory  on  26-29 
January  1999.  This  special  meeting  explored 
ways  to  ensure  modeling  and  simulation  (M&S) 
credibility  in  effective,  efficient,  and  affordable 
ways.  The  SIMVAL  99  Workshop  was  co¬ 
sponsored  by  the  Military  Operations  Research 
Society  (MORS)  and  the  Society  for  Computer 
Simulation  (SCS)  International,  and  was  the 
fourth  in  the  series  of  SIMVAL  workshops 
sponsored  by  MORS.  SIMVAL  99  examined 
current  technology  supporting  verification, 
validation  and  accreditation  (W&A)  and 
identified  promising  areas  of  possible  technology 
development  that  could  enhance  W&A.  Such 
enhancements  could  make  W&A  more  effective 
for  a  given  level  of  resources  or  could  allow  the 
same  level  of  W&A  to  be  achieved  with  reduced 
resources.  The  SIMVAL  99  Workshop  focused 
on  three  specific  areas:  (1)  verification 
technology,  (2)  validation  technology  and 
methodology  and  (3)  the  impact  of  technology  on 
W&A  costs.  The  primary  objectives  of 
SIMVAL  99  were  to  provide  a  synopsis  of  the 
state  of  the  art  for  W&A  technology  and  to 
suggest  what  should  be  done  to  most  fully  exploit 
that  technology  and  advance  W&A. 


The  SIMVAL  99  report  has  three  parts.  1)  the 
report  body  which  contains  the  findings 
/observations  and  recommendations  of  the 
workshop  as  a  whole  and  of  the  individual  groups 
within  the  workshop;  2)  the  appendices  of  the 
report  which  contain  supplementary  information 
about  the  workshop  (its  attendance,  portions  of 
the  read-ahead  and  handout  materials,  etc.)  and  3) 
an  annex  which  contains  the  briefings  presented 
at  SIMVAL  99  in  its  opening  and  closing  plenary 
sessions  as  well  as  the  briefings  presented  within 
the  three  SIMVAL  99  working  groups  (most  of 
the  briefings  are  not  annotated).  All  three  of 
these  items  are  available  from  the  MORS  Office 
and  from  DTIC;  the  report  body  and  the 
appendices  of  the  report  will  be  available  on  the 
MORS  website  as  the  report  of  SIMVAL  99. 
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Introduction 


Since  the  late  1980s,  MORS  has  led  conceptual 
development  for  simulation  validation 
(SIMVAL).  Since  the  last  SIMVAL  workshop 
(held  in  1994),  Verification,  Validation  and 
Accreditation  (VV«&A)  have  become  central 
themes  in  Department  of  Defense  (DoD) 
modeling  and  simulation.  DoD  Instruction 
5000.61,  released  in  April  1996,  set  Defense 
policy  guidance  for  W&A.  The  DoD  WStA 
Recommended  Practices  Guide  (RPG),  published 
in  November  1996,  provides  W&A  guidance 
within  DoD  today.  Efforts  are  underway  to 
improve  the  RPG  by  incorporating  lessons 
learned  from  those  who  have  followed  its 
guidance.  These  developments  tmderscore  the 
need  for  advancing  the  state-of-the-art  in  W&A. 
Decision  makers,  program  managers,  simulation 
developers,  V&V  practitioners,  W&A  agents, 
V&V  theorists,  academe  and  simulation  users  are 
all  concerned  with  the  conduct  of  W&A.  The 
full  scope  of  ciurent  W&A  technology  needs  to 
be  better  understood.  Such  exposure  of  W &A 
technologies  should  include  not  only  tools  and 
methodologies  for  performing  V&V,  but  should 
also  address  costs  associated  with  conducting 
W&A.  This  process  should  identify 
technologies  which  can  increase  V&V 
effectiveness  without  increasing  costs  or  decrease 
costs  without  degrading  V&V  effectiveness. 

W&A  technologies  and  methodologies  have 
pertinence  for  those  outside  DoD  as  well  as  for 
those  within  the  Defense  community.  Co¬ 
sponsorship  of  the  SIMVAL  Workshop  99  by  the 
Society  for  Computer  Simulation  (SCS) 
International  facilitated  involvement  in  SIMVAL 
99  by  those  outside  the  Defense  community  and 
enlarged  the  reach  of  insights  from  SIMVAL  99 
beyond  the  Defense  community.  SIMVAL  99 
included  participants  from  outside  the  US 
(Australia,  Germany  and  the  Netherlands)  and  a 
few  displays/exhibits  of  V&V  technology  for 
some  of  the  tools  identified  in  Appendix  F. 

The  report  body  begins  with 
findings/observations  and  recommendations  of 
SIMVAL  99  as  a  whole.  The  structure  and 
process  used  in  the  conduct  of  this  meeting  are 
then  described  in  the  following  paragraphs. 
Synopses  from  the  three  Working  Groups  and  the 


Synthesis  Group  are  also  included.  These 
synopses  help  to  clarify  current  and  future  states 
of  W&A  technology.  The  Working  Group  (WG) 
and  Synthesis  Group  (SG)  synopses  include 
findings/observations  and  recommendations, 
some  of  which  are  not  identical  with  overall 
SIMVAL  99  findings  and  recommendations.  This 
situation  exists  because  the  findings  and 
recommendations  of  the  synopses  are  from  the 
particular  perspective  of  the  group  while  the 
overall  findings  and  recommendations  balance 
perspectives  from  the  whole  W&A  community. 
The  body  of  this  report  ends  with  a  brief 
conclusion  and  assorted  comments.  The 
following  materials  are  contained  in  appendices: 


Appendix  A: 

Appendix  B: 
Appendix  C: 


Appendix  D: 


Appendix  E: 
Appendix  F: 


SIMVAL  99  Terms  of  Reference 
(TOR) 

SIMVAL  99  Agenda 
SIMVAL  99  Participants 
(including  exhibitors  and  WG 
assignments) 

Identification  of  SIMVAL  99 
presentations  (plenary  keynote 
session  briefings,  materials 
presented  in  working  groups  and 
the  SG  and  Thursday  afternoon 
outbriefs)  -  actual  briefings  are 
contained  in  the  SIMVAL  99 
Aiuiex. 

Acronyms 

Read-ahead  and  Selected 
Handout  Items 


Disclaimer  of  Endorsement;  Reference  in 
SIMVAL  99  materials  to  a  specific  commercial 
product,  process  or  service  by  trade  name, 
trademark,  manufacturer  or  otherwise,  does  not 
constitute  or  imply  its  endorsement, 
recommendation,  or  favoring  by  the  United  States 
Government,  by  MORS,  by  SCS  International  or 
by  any  other  organization  or  person.  A  number  of 
commercial  and  government  tools  and  software 
packages  are  specifically  identified  in  this  report. 
They  are  mentioned  as  representative  of  particular 
capabilities.  The  items  identified  are  ones  which 
are  known  by  SIMVAL  99  participants  and  do  not 
necessarily  reflect  the  totality  of  such  tools  and 
software  packages  which  may  be  available. 
Likewise,  capabilities  asserted  for  such  tools  and 
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software  packages  are  merely  the  perception  of 
SIMVAL  99  participants  and  should  not  be 
accepted  as  fact  simply  on  the  basis  of  material  in 
this  report. 
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SIMVAL  99  Findings/Observations  and  ReconMnendations 


1.  Findings  &  Observations 

The  modeling  and  simulation  (M&S)  community, 
and  the  subset  of  people  within  it  who  are 
concerned  with  verification,  validation  and 
accreditation  (W &A),  are  so  broad  and  diverse 
that  effective  communication  is  very  difficult. 
Commonly  accepted  definitions  for  W&A  terms, 
such  as  those  provided  within  the  DoD  M&S 
glossary  and  in  formal  DoD  documents  such  as 
DoD  directives  and  the  DoD  WScA 
Recommended  Practices  Guide  (RPG),  can  be 
(and  are!)  interpreted  in  such  different  ways  by 
various  members  of  the  M&S  community  that 
misunderstandings  and  communication  confusion 
can  abound.  Lack  of  a  clear  articulation  of  the 
various  perspectives  that  can  be  brought  to  M&S 
W&A  is  part  of  the  cause  for  this  problem. 

Those  working  with  software  architectures  have 
recognized  the  importance  of  multiple  views  for 
the  architecture  of  a  software  development  to 
provide  a  full  description  of  the  planned 
development,  such  as  the  operational,  system  and 
technical  views  espoused  in  the  DoD  Joint 
Technical  Architecture  (JTA).  M&S  W&A  have 
not  matured  to  the  place  of  having  formalized  the 
different  perspectives  that  are  equivalent  to  the 
different  views  in  software  architecture.  Also 
contributing  to  communication  ambiguity  in  M&S 
W&A  is  the  lack  of  a  mathematical  level  of 
specificity  for  most  W&A-related  definitions. 
This  problem  manifested  itself  significantly  in 
WG  1  and  impeded  their  progress  until  it  was 
decided  for  the  group  to  split  and  tackle 
verification  issues  independently  from  a  global 
perspective  and  from  a  phase-specific  verification 
process  perspective.  It  is  believed  that  the 
breadth  and  depth  of  potential  trouble  that  such 
communication  difficulties  can  cause  the  M&S 
W&A  community  are  not  fully  appreciated  by 
either  W&A  practitioners  or  M&S  management. 
It  appears  that  the  W&A  community  is  not 
exploiting  existing  technology  as  much  as  desired. 
The  reasons  for  this  are  manifold.  First,  M&S 
management  and  W&A  practitioners  as  a  whole 
are  woefully  unaware  of  existing  tools  and 
technologies  that  could  be  used  to  support 
W&A.  Second,  the  W&A  community  has 
focused  primarily  to  date  on  defining  terminology 
and  developing  methodologies  and  processes,  and 


has  not  given  adequate  attention  to  the  potential 
benefits  of  tools  and  technologies.  Other  reasons 
include  the  lack  of  a  comprehensive  survey  of 
tools  and  technologies  available  to  support  the 
education  of  the  W&A  community  or  the  use  of 
these  resources  in  DoD  and  elsewhere.  No 
central  repository  exists  to  document  W&A  tool 
use  or  to  serve  as  a  resomce  for  future 
applications  of  W&A  tools  and  technologies. 
Consequently,  resources  to  support  W&A  tool 
use  are  not  identified  routinely  as  part  of  M&S 
lifecycle  planning.  Even  when  tools  are  used, 
their  use  is  often  ad  hoc  and  not  repeated 
consistently  from  one  M&S  project  to  the  next. 

It  appears  that  advancement  in  computational 
capabilities  and  software  engineering  are 
proceeding  more  rapidly  than  comparable 
advances  in  M&S  and  W&A.  This  problem  is 
exacerbated  by  the  limited  awareness  of  M&S 
management  and  W&A  practitioners  about 
potential  current  and  evolving  technology  to 
facilitate  more  effective  and  affordable  W&A. 
However,  it  appears  that  the  time  may  be 
appropriate  for  major  advances  in  application  of 
M&S  W&A  technology.  Increased  emphasis  is 
being  placed  upon  M&S  use  in  system  design, 
testing,  and  acquisition,  as  illustrated  by  the 
Simulation  Based  Acquisition  (SBA)  and  Joint 
Experiment  (JE)  programs,  and  upon  model-based 
decision  aids  for  operational  planning  and 
execution.  The  Department  of  Energy  (DOE) 
Accelerated  Strategic  Computing  Mtiative 
(ASCI)  program  includes  a  major  software 
development  component  (with  some  emphasis  on 
V&V  of  codes  used  in  computational  science  and 
other  areas  of  “grand  challenge”  problems)  as 
well  as  the  ASCI  emphasis  on  advances  in 
computational  capabilities.  Continuing 
computational  advances  and  improvements  in 
software  development  environments  make 
automation  of  M&S  V&V  more  viable  just  as 
they  have  already  demonstrated  such  automation 
viability  in  software  V&V.  However,  it  should  be 
noted  that  procurement  officials,  project  managers 
and  planners  also  have  to  be  aware  of  the  benefits 
of  leveraging  tools  used  in  M&S  development  for 
V&V  efforts  so  that  contracts,  etc.  will  be  written 
to  effectively  accommodate  exchange  and  sharing 
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of  information,  data  and  tools  necessary  for  these 
benefits  to  be  realized. 

W&A  tools  and  technologies  should  be  able  to 
reduce  program  cost  and  risk  if  used  properly. 
However,  the  importance  of  making  the  necessary 
investment  to  enable  use  of  such  M&S  W&A 
tools  and  technologies  has  not  been  widely 
recognized  within  DoD  or  elsewhere.  In  part,  this 
situation  exists  because  there  is  very  little  reliable 
cost  benefit  information  available  relative  to 
M&S  W&A.  Available  cost  information  is  not 
well  organized  for  effective  analysis  (no  standard 
cost  element  identification  exists,  useful  and 
widely  accepted  metrics  do  not  exist,  etc.).  This 
reduces  the  basis  for  cost  benefit  assessments  to 
either  mere  postulation  or  to  argument  by  analogy 
through  comparisons  between  the  dearth  of  tool 
use  to  support  M&S  W&A  with  the  more 
extensive  application  of  automated  tools  in  the 
software  V&V  arena. 

It  appears  that  commercial  forces  will  continue  to 
drive  the  development  of  software  V&V 
technology  more  rapidly  than  M&S  management 
and  W&A  personnel  are  likely  to  assimilate  and 
exploit  fully.  However,  that  may  not  be  the  case 
for  technology  that  is  peculiar  to  M&S  W&A 
that  is  otherwise  not  generally  applicable  for 
software  V&V.  The  stimulus  of  government 
research  and  development  (R&D)  funding  may  be 
required  for  needed  progress  in  this  area. 

A  theme  that  was  prominent  in  many  parts  of  the 
workshop  is  the  importance  of  talented  people. 
Tools  and  technology  can  supplement,  but  will 
not  replace  the  important  role  that  knowledgeable 
people  have  in  effective  W&A.  Advancement  in 
M&S  and  V&V  theory  can  lessen  dependence 
upon  the  most  talented  people  as  standardized 
formalisms  (which  theoretical  advances  enable) 
can  be  employed  successfully  by  a  larger  group  of 
people  to  W&A  problems. 

2.  Recommendations 

There  needs  to  be  a  sustained  educational 
campaign  to  ensure  that  M&S  managers 
appreciate  the  importance  of  W&A,  understand 
the  cost-benefit  potential  of  W&A  for  their 
applications  and  understand  the  resources 
required  to  exploit  W&A  technology  in  ways 


that  are  both  effective  and  affordable.  This 
educational  campaign  must  also  ensure  that 
W&A  practitioners  are  aware  of  available 
methodologies,  techniques,  tools,  technologies 
and  the  implications  of  their  application  or  non¬ 
application.  No  single  organization  is  likely  to 
stimulate  or  provide  the  total  impetus  needed  for 
such  an  educational  campaign;  instead  each  of  the 
MORS  Sponsors,  SCS  leadership,  DMSO  and 
others  are  encouraged  to  establish  educational 
campaigns  within  their  spheres  of  influence  or 
provide  the  necessary  resources  to  allow  their 
people  to  benefit  from  the  educational  campaigns 
of  other  organizations.  Such  educational 
campaigns  must  be  multi-faceted,  not  only 
addressing  audiences  of  W&A  practitioners 
through  short-courses,  conference  presentations 
and  articles  in  peer-reviewed  trade  journals,  but 
also  addressing  M&S  managers. 

Several  areas  of  research  are  required.  At  the 
most  fundamental  level,  a  relatively 
comprehensive  survey  of  available  W&A  tools 
and  technologies  is  required.  This  survey  should 
address  how  such  tools  and  technologies  should 
be  used  properly,  identify  appropriate 
applications  and  link  tool  use  to  phases  of  the 
M&S  lifecycle.  Information  in  this  report  can 
contribute  to  that  survey,  provide  clues  about  how 
the  information  might  be  structured  and  pointers 
to  some  of  its  locations.  Information  resource 
requirements  for  tool  use  is  also  needed,  such  as 
data,  personnel,  training,  equipment  and  financial 
considerations.  Research  should  include 
establishment  of  a  repository  of  W&A  tool 
information  that  identifies  prior  tool  uses,  results 
of  tool  application  and  costs  of  use.  This 
repository  may  be  either  new  or  included  within 
an  existing  repository.  In  addition,  some  level  of 
support  for  basic  research  related  to  W&A  is 
indicated.  This  arena  has  been  neglected  to  date, 
and  such  investment  is  needed  if  the  long-term 
promise  of  W&A  improvements  from 
technology  advancements  is  to  be  fully  realized. 
The  development  of  a  formalized  methodology  is 
also  needed  to  assist  users  in  determining  how 
good  a  simulation  must  be  if  it  is  to  support  a 
particular  kind  of  application.  When  such 
guidance  can  be  reduced  to  formalized 
methodology,  it  becomes  amenable  to  automation, 
with  the  consequence  of  enhanced  W&A 
effectiveness  concurrent  with  reduction  in  the 
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cost  of  performing  W&A.  In  addition  to  DoD 
investments  in  research,  agencies  external  to  DoD 
that  support  basic  M&S  research,  such  as  the 
National  Science  Foundation  (NSF),  other 
government  agencies,  commercial  enterprises  or 
private  foundations,  should  consider  the  urgent 
need  for  more  rapid  advancement  of  W&A 
methodologies  and  technologies  when  dispersing 
their  funds.  A  corollary  to  the  above  is  the 
necessity  for  sharing  information  developed  by 
such  research  throughout  the  M&S  W&A 
community 

Development  of  a  widely  accepted  and  widely 
used  way  to  identify  and  describe  different 
W&A  perspectives  is  also  needed  to  reduce 
conununication  confusion  among  disparate 


elements  of  the  M&S  and  W&A  communities. 
Concomitant  with  development  of  such  standard 
perspectives  (views)  would  be  elaboration  of 
W^&A-related  terminology  to  provide  greater 
precision  of  meaning.  Specification  of  W&A 
cost  elements  must  be  included  in  this  work,  so 
that  meaningful  W&A  cost  data  can  be  captured 
to  provide  a  factual  basis  for  development  or 
refinement  of  W&A  cost  models. 

In  addition  to  the  above  general 
recommendations,  the  M&S  W&A  community 
should  also  give  attention  to  the  more  specific 
recommendations  contained  within  the  synopses 
from  the  three  woridng  groups  and  the  Synthesis 
Group. 
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SIMVAL  99  Structure  and  Process 


SIMVAL  99  employed  plenary  sessions  to  start 
the  workshop,  to  facilitate  cross-fertilization  of 
ideas  among  its  several  groups  each  day,  and  to 
provide  for  communication  of  insights  at  the  end 
of  the  workshop  to  all  participants.  In  the 
opening  plenary  session,  CAPT  Dennis  McBride 
(Office  of  Naval  Research,  Program  Manager  for 
Medical  S&T)  brought  the  keynote  address, 
which  challenged  SIMVAL  99  participants  to 
discover  ways  to  keep  W&A  from  being 
outpaced  by  the  speed  of  general  coiiq)utation  and 
software  advance.  Col  Kenneth  Konwin,  Director 
of  the  Defense  Modeling  and  Simulation  Office 
(DMSO),  continued  the  opening  plenary  session 
with  a  V&V  technology  to  calibrate  SIMVAL  99 
participants  about  current  technology  capabilities. 
A  cautionary  perspective  on  W&A  from  Dr. 

Paul  Davis  (RAND)  closed  the  opening  plenary 
sessions.  He  noted  issues  of  research  base, 
bureaucratic  mischief  that  impedes  high-quality 
analysis,  and  opposing  trends  in  commercial  off- 
the-shelf  software,  some  of  which  help  and  some 
of  which  hurt  the  cause  of  W&A,  and  suggested 
principles  for  W&A  related  tools,  illustrating 
some  of  the  points  with  examples  from  recent 
work.  Several  displays  and  exhibits  about  V&V 
technology  and  tools  were  available  for  SIMVAL 
99  participants  on  the  first  day  of  the  workshop. 

The  workshop  attendees  were  divided  into  three 
working  groups  (identified  below).  The  working 
groups  occasionally  subdivided  to  accomplish 
dieir  responsibilities  more  expeditiously.  A 
Synthesis  Group  facilitated  communication  of 
significant  points  among  the  working  groups  and 
assisted  the  SIMVAL  99  co-chairs  in  managing 
the  workshop.  More  information  about  SIMVAL 
99’ s  organization,  leadership,  agenda  and 
processes  may  be  found  in  the  SIMVAL  99  Terms 
of  Reference  (TOR)  in  Appendix  A. 

1.  Working  Group  1;  Verification 
Tftphnnlogy:  Topics  which  WG  1  addressed 
included  automated  requirements/specification¬ 
testing  and  CASE  tools,  as  they  are  used  in  the 
context  of  modem  simulation  paradigms  (object 
oriented  based  simulation,  distributed  simulation, 
extensive  use  of  Artificial  Intelligence  (AJ)  and 
other  adaptive  methodologies,  etc.).  Descriptive 
formats  and  their  interactions  (e.g.,  this  is  where 


data/meta-data  issues  would  reside)  is  a  subset  of 
verification  technology.  The  emphasis  of  this 
WG  was  on  identification  of  available  tools  and 
their  applicability,  especially  to  military  uses  of 
M&S. 

2.  Working  Group  2:  Validation  Technology 
and  Methodol<^:  WG  2  focused  on  applications 
of  technology  to  support: 

a.  Subject  Matter  Expert  (SME)  selection, 
orientation,  management,  use,  etc., 
especially  in  conceptual  validation 
processes  (and  also  in  results  validation) 

b.  Correlation  of  representational  fidelity  with 
application  needs 

c.  Criteria  for  establishing  needed  levels  of 
simulation  fidelity 

d.  Automation/technology  potential  (e.g., 
animation)  to  support  validation 
methodology 

e.  Correlation  among  the  various  levels  of 
simulation  CMs  (CMMS,  simulation 
concept,  simulation  elements/sub-elements) 

3.  Working  Group  3:  Impact  of  Technology 
on  W&A  Costs:  WG3  addressed  identification 
of  W&A  cost  elements,  costing  algorithms,  and 
mles-of-thumb. 

4.  Synthesis  Group:  The  SG  provided  a 
mechanism  to  ensure  cross-fertilization  of  ideas 
among  the  working  groups,  and  to  integrate  and 
synthesize  ideas  from  the  workshop.  Members  of 
the  SG  participated  in  working  groups  as  a  means 
of  facilitating  conceptual  synthesis  and 
integration. 

Of  the  115  people  who  participated  in  SIMVAL 
99  (listed  in  Appendix  C):  33  were  assigned  to 
WG  1  (Verification  Technology),  35  to  WG  2 
(Validation  Methodology  and  Technology),  16  to 
WG  3  (Impact  of  Technology  on  W&A  Costs), 
and  7  to  the  SG.  Just  a  few  people  changed 
working  groups  during  the  workshop.  Members 
of  the  Synthesis  Group  participated  in  the 
working  groups  as  needed  to  facilitate  cross¬ 
fertilization  of  ideas  among  the  groups  and  to  help 
the  SIMVAL  99  Co-Chairs  keep  the  working 
groups  focused  on  their  assigned  responsibilities. 
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Thirteen  attendees  participated  only  in  the 
opening  plenary  session  of  the  first  day.  The 
remainder  consisted  of  the  workshop  leadership 
(SIMVAL  99  Co-Chairs)  and  support  team  fi*om 
the  MORS  office  and  JHU/APL. 

SIMVAL  99  had  a  reasonable  balance  in  the 
organizational  representation  of  its  participants, 
as  indicated: 


Participants 


Military 

14 

Army 

5 

Navy 

5 

Air  Force 

4 

Government  Civilian 

25 

FFRDC/UARC 

22 

Lidustry 

41 

Academe 

5 

Non-US 

3  (from  Australia, 
Germany,  and  The 
Netherlands 
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SD^AL  99  Working  Group  and  Synthesis  Group  Synopses 


The  following  pages  provide  synopses  from  each 
of  the  three  SIMVAL  99  working  groups  and 
from  its  SG.  These  materials  provide  more 
detailed  insights  about  findings  and  observations 
than  presented  earlier  in  the  report.  These 
synopses  also  contain  recommendations,  some  of 
which  are  not  the  same  as  recommendations 
presented  earlier  in  this  report  since  these 
recommendations  are  from  the  particular 
perspective  of  the  group.  The  earlier 
recommendations  balanced  perspectives  of  the 
groups  and  of  the  larger  M&S  community  with  its 
W&A  components. 


Leadership  and  participants  of  the  groups  are 
identified  at  the  beginning  of  each  synopsis  so 
that  the  reader  can  appreciate  the  breadth  and 
depth  of  experience  and  associations  which 
contributed  to  the  materials  in  the  s)mopses. 
Additional  information  may  also  be  found  in 
Appendix  D  about  the  briefings  presented  to  the 
working  groups  and  in  the  SIMVAL  99  Annex 
which  contains  both  the  briefings  presented  to  the 
groups  and  their  outbriefs  to  the  workshop  (note: 
most  of  these  briefings  are  not  annotated). 
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SIMVAL  99  Working  Group  1  (Verification  Technology)  Synopsis 

Working  Group  1  Leaders: 

Co-Chairs:  Dr.  Osman  Balci  (Virginia  Tech) 

Mr.  Robert  Lewis  (Tec-Masters,  Inc.) 

Reporter:  Ms.  Susan  Solick  (US  Army  TRAC) 


Participants  (33  including  WG 1  leaders): 


Arthur,  DR  James 

Virginia  Tech 

Brasse,  MR  M.H.H. 

TNO  Physics  and  Electronics  Lab  (The 
Netherlands) 

Buckley,  MR  Joseph 

Lockheed  Martin-Government  Electronics 
Systems 

Cantwell,  MR  Larry 

US  Army  TRACDOC  Analysis  Center 

Chambers,  MR  Joseph  Steve 

Space  and  Missile  Defense  Battle  Lab 

Christensen,  MR  Peter 

The  MITRE  Corporation 

Crooks,  MR  Mark 

RAND 

Dickerson,  MS  Theresa 

CAS  Inc. 

Eirich,  MR  Peter 

Johns  Hopkins  University/APL 

Gibson,  DR  Forrest 

Joint  National  Test  Facility 

Haddix,  MR  F.  Furman 

University  of  Texas  at  Austin 

Haeerdon,  MS  Linda 

Space  &  Missile  Command 

Iwatake,  MR  Brian 

Naval  Undersea  Warfare  Center  (NUWC) 

Jordan,  MR  Jack 

BMH  Associates 

Kollmorgen,  MR  Gary 

BMH  Associates,  Inc 

Leake,  DR  Charles 

JWARS  Office 

Mair,  MR  Hans 

IDA 

Martin,  MR  Michael 

Naval  Air  Warfare  Center 

Martin,  MS  Terri  L  Schrimsher 

us  Army  Space  and  Missile  Defense  Battle 

Lab 

Michelsen,  Randy 

Los  Alamos  National  Laboratory 

Mihaloew,  MR  Reed 

The  MITRE  Corporation 

Nance,  DR  Richard 

Virginia  Tech 

Prosser,  MR  Terry 

Logicon 

Ravenel,  MRS  Cynthia 

USA  Aviation  and  Missile  Command 

Sargent,  DR  Robert 

Syracuse  Uniyersity 

Senko,  MR  Robert 

Delfin  Systems  (DMSO) 

Smith,  Capt  Sandra 

JADS  Joint  Test  Force 

Soriesterbach,  MR  Thomas 

Johns  Hopkins  University/APL 

Strickland,  MR  Edward 

Analex  Corporation 

Wallace,  MS  Dolores 

US  Department  of  Commerce 

This  synopsis  has  three  parts.  The  first  part 
identifies  the  invited  presentations  to  WG  1 
during  SIMVAL  99.  Content  of  these 
presentations  is  included  in  the  SIMVAL  99 
Annex.  Because  WG  1  functioned  as  two  sub¬ 


groups  for  part  of  the  workshop,  each  of  which 
produced  a  synopsis  of  ideas  about  verification 
technology,  it  was  decided  to  provide  these  two 
synopses  as  the  second  and  third  parts  of  this 
synopsis  for  WG  1  since  there  was  not  adequate 
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time  during  the  workshop  to  produce  a  single, 
fully  integrated  synopsis  that  properly  represented 
the  perspectives  of  both  subgroups.  This 
approach  provides  the  reader  wiA  a  fuller 
appreciation  for  topics  addressed  during  the 
workshop,  and  provides  two  complementary 
perspectives  about  current  and  future  verification 
technology. 

1.  WG 1  Invited  Presentations 

The  invited  presentations  are  listed  in  the  order  in 
which  they  were  actually  presented.  It  had  been 
planned  originally  for  them  to  be  presented  in  the 
order  indicated  by  the  numbers  in  parentheses 
(just  in  front  of  the  presentation  title)  so  that  a 
“from  beginning  to  end”  perspective  about 
verification  processes  could  be  obtained.  The 
arrangement  of  these  presentation  materials  in  the 
SIMVAL  99  Annex  follows  the  order  originally 
platmed. 

(1)  "M&S  Verification  Technology:  An 
Overview"  by  Osman  Balci:  to  identify  the 
issues  and  problems  to  be  addressed  during 
the  workshop. 

(7)  "Overview  of  Six  Major  W&A  Processes 
and  Associated  Cost  Models"  by  Robert  O. 
Lewis  to  a  combined  session  for  Working 
Groups  1  (Verification  Technology)  and  3 
(W&A  Costs) 

(6)  "M&S  Implementation  Verification"  by 
Richard  E.  Nance 

(5)  "M&S  Design  Verification"  by  Delores  R. 
Wallace 

(2)  "M&S  Requirements  Verification"  by  Gary 
KoUmorgen 

(3)  "M&S  Requirements  Verification"  by  Randy 
Michelsen 

(4)  "M&S  Conceptual  Model  Verification"  by 
Furman  Haddix 

2.  Synopsis  from  Subgroup  A  of  WG  1 
a.  Topics  Addressed: 

•  Findings/observations  on  the  State  of 
Verification  Technology 

•  Findings/observations  on  the  Near  Term 
Potential  for  Verification  Technology 

•  Findings/observations  on  the  Far  Term 
Potential  for  Verification  Technology 

•  Issues 


b.  Findings  and  Observations  on  the 

Current  State  of  Verification  Technology 

•  Automation  is  critically  needed  to 
effectively  and  affordably  conduct  M&S 
verification. 

•  Component-based  software  development  is 
an  effective  and  affordable  way  of  creating 
M&S  applications  and  conducting  M&S 
verification.  A  verified  model  component 
can  substantially  decrease  the  M&S 
verification  effort  when  reused 
appropriately  —  thereby  significantly 
decreasing  the  time  and  cost  of  M&S 
development. 

•  Success  in  the  application  of  the  current 
state  of  the  verification  technology  is  very 
much  affected  by  employment  of  skilled 
persoimel.  Education  and  training  in  the 
area  of  technology  for  M&S  development, 
especially  in  regard  to  W&A,  appears  to 
be  less  than  what  is  needed  to  fully  exploit 
current  capabilities. 

•  Terminology  continues  to  pose  serious 
communication  problems  for  M&S  and 
W&A  practitioners  and  others  due  to  the 
diversity  of  the  M&S  community. 
Definitions  of  terms,  even  when  agreed 
upon,  may  be  interpreted  differently, 
incorrectly,  liberally  or  loosely.  Such 
inconsistent  use  of  M&S  verification 
terminology  adversely  affects  the 
development,  use  and  dissemination  of  the 
verification  technology. 

•  M&S  verification  technology  heavily 
relies  on  software  verification  technology. 
Advancements  in  M&S  verification 
capabilities  mainly  come  from  the 
software  engineering  discipline.  This 
requires  M&S  verification  professionals  to 
be  knowledgeable  about  software 
verification  technology  and  associated 
CASE  tools  so  that  the  software  aspects  of 
M&S  verification  can  be  addressed 
effectively. 

•  Lack  of  journal-quality  peer-reviewed 
publications  is  a  serious  problem,  which 
hinders  advancement  and  dissemination  of 
information  about  M&S  verification 
technology.  Incentives  or  encouragement 
for  peer-reviewed  publication  within  either 
the  government  or  industry  M&S 
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communities  seem  to  be  lacking.  This 
problem  manifests  itself  in  the  following 
example.  Neither  the  W&A  Area  Editor 
of  the  ACM  Transactions  on  Modeling  and 
Computer  Simulation  nor  the  Validation 
Associate  Editor  for  Simulation  have 
received  submission  of  papers  specifically 
addressing  W&A  during  the  last  two 
years. 

•  Significant  advancement  (in  the  near  term) 
of  verification  technology  specifically 
oriented  toward  M&S  development  and 
application  is  likely  to  be  achieved  only  if 
substantial  funding  is  provided  for  this 
purpose  (as  a  stimulus)  by  government 
agencies  such  as  DARPA,  Office  of  Naval 
Research  (ONR)  and  DMSO. 

•  Current  verification  technology  is  very 
rich  in  terms  of  non-automated  techniques, 
many  of  which  can  be  employed  for  M&S 
verification  throughout  the  entire  M&S 
development  life  cycle.  Such  techniques 
are  described  in  contemporary  W&A 
literature,  such  as  those  identified  in  the 
SIMVAL  99  Terms  of  Reference  (TOR), 
read-ahead  package  and  handout  materials. 

c.  Findings  and  Observations  on  the  Near 
Term  Potential  for  Verification 
Technol<^ 

•  Many  Computer-Aided  Software 
Engineering  (CASE)  tools  are  available 
commercially  that  could  be  used  to 
provide  much  of  the  automation  needed  in 
M&S  verification.  This  automation  is 
needed  for  more  effective  and  more 
affordable  M&S  verification.  Exploitation 
of  these  capabilities  requires  that  M&S 
verification  personnel  become  familiar 
with  available  CASE  tools  and  use  them  as 
much  as  possible  —  it  also  requires  that 
M&S  management  make  necessary 
investment  in  procurement  of  such  tools 
and  personnel  training  for  their  use. 

d.  Findings  and  Observations  on  the  Far 
Term  Potential  for  Verification 
Technology 

•  Component-based  M&S  development 
technology,  when  created,  may  be  the 
“silver  bullet”  which  enables  both 
effective  and  affordable  M&S  verification. 


•  Development  of  software  tools  to  provide 
automation  specifically  for  M&S 
verification  should  result  in  more  effective 
and  more  affordable  M&S  verification. 

•  Availability  of  more  appropriately 
educated  and  trained  VV&A  personnel 
should  result  in  more  successful 
applications  of  M&S  verification 
technology. 

3.  Issues 

a.  Is  tiiere  a  lack  of  identifiable,  qualified 
V&V  personnel? 

•  Discussion: 

•  Q:  How  does  this  differ  from  the  lack  of 
software  engineers  (e.g.,  engineers, 
managers  and  practitioners)?  It’s  the  same 
problem. 

•  Q;  Is  there  really  a  shortage?  Or  can  we 
just  not  identify  the  right  people?  Is  it  the 
skills  or  the  bodies  we  are  talking  about? 
There  is  very  little  agreement  on  what 
criteria  is  necessary  regarding  training, 
education  or  skills  necessary  for  M&S  (or 
M&S  V&V). 

•  Q;  Are  we  talking  M&S  or  V&V?  We 
need  to  identify  appropriate  M&S  people 
first.  The  lack  of  skill  in  simulation  is 
why  we  have  a  lack  in  skill  in  the  V&V 
arena.  The  issue  is  the  identifiability  of 
qualified  people.  Organizations  are 
assigning  unskilled  or  inappropriately 
skilled  people  to  acconq)lish  this  important 
task.  There  is  no  job  in  the  M&S  or  V&V 
category  in  many  organizations.  This 
leads  to  a  lack  of  training  in  these  areas. 

b.  Should  there  be  a  qualify  assurance  (QA) 
process?  If  so,  then  should  it  include 
V&V? 

•  Discussion: 

•  QA  applied  to  M&S  examines  quality 
features  of  the  model  itself.  The  focus  in 
M&S  V&V  has  been  on  validation  rather 
than  verification.  This  is  why  “QA”  is 
M&S  mission  oriented. 

•  Q:  Should  V&V  be  de-emphasized  and 
QA  be  given  more  emphasis  (i.e.  keep 
V&V  back  in  the  software  development 
process  where  it  started)?  Both  are 
needed.  QA  focuses  upon  process  and  is 
normally  performed  exclusively  by  the 
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M&S  developer;  V&V  focuses  upon  the 
product  and  may  be  performed  in  part  by 
those  who  are  not  part  of  the  M&S 
development  team,  increasing  the  potential 
for  increased  objectivity. 

c.  Should  methods  drive  tools  ratiher  than 
tools  driving  methods? 

•  Discussion: 

•  It  happens  the  other  way  around  too  often. 

•  Q:  Shouldn’t  the  objectives  drive  the 
methodology? 

•  Picking  tools  for  each  phase  is  not 
optimal.  We  should  try  to  capture  the  idea 
of  developing  a  V&V  methodology  and 
identifying  an  environment  to  support  that 
methodology. 

•  Recommendation: 

•  W&A  literature,  such  as  the  DMSO  RPG 
for  W&A,  needs  to  identify  appropriate 
V&V  methodologies  for  each  phase  of  the 
M&S  lifecycle  and  M&S  development  and 
application  process. 

d.  Should  a  uniform  V&V  procedure  he 
applied  to  all  M&S  projects?  (i.e.,  One 
size  fits  all?) 

•  Discussion: 

•  There  are  many  variables  that  influence 
the  amount  and  kind  of  V&V  that  should 
be  applied  to  an  M&S  project.  These 
include  the  type  and  size  of  the  M&S,  its 
intended  use,  tools  used  in  developing  the 
M&S,  fidelity  expected  of  the  M&S, 
maturity  of  the  M&S  project  and  its 
developer,  etc.  Thus,  it  should  be  obvious 
that  one  size  of  V&V  does  not  fit  all  M&S 
projects. 

•  Recommendation: 

•  The  level  of  the  V&V  effort  should  be 
determined  by  appropriate  consideration 
of  factors  such  as  those  mentioned  about 
and  also  be  commensurate  with  the  level 
of  the  overall  M&S  development  effort, 
which  may  include: 

—  Developer  V&V 
—  Combined  developer/customer  V&V 
-  Independent  V&V  (Note: 

“Independent”  should  not  be  construed 
as  “isolated.”) 


e.  Should  there  he  more  training  and 
education  provided  for  W&A?  (short 
courses,  university  programs,  etc.) 

•  Recommendation: 

Such  training  and  education  should  be 
M&S-oriented  and  not  solely  oriented 
toward  W&A  in  isolation  —  M&S 
training  and  education  should  always 
include  W&A  as  part  of  its  curriculum. 

f.  Should  there  he  more  funding  resources 
for  basic  research  in  M&S  and  W&A? 

•  Discussion: 

Resources  identified  in  the  SMVAL  99 
TOR  note  that  M&S  progress  is  impeded 
by  the  lack  of  generally  accepted, 
comprehensive  theory  for  M&S 
development  (and  its  elements,  such  as 
W&A). 

g.  Should  M&S  development  methodology 
he  better  integrated  with  the  W&A 
processes? 

•  Recommendation: 

As  a  whole,  the  M&S  community  needs  to 
do  a  better  job  of  following  the  guidance 
in  this  area  provided  by  the  DMSO  RPG 
for  W&A  and  other  contemporary 
W&A  publications. 

h.  Should  increased  awareness  of  W  &  A 
throughout  the  M&S  community  be 
made  a  priority? 

•  Discussion: 

The  M&S  community  at  large  has  limited 
understanding  of  W&A  and  related 
issues. 

•  Recommendation: 

The  MORS  Sponsors,  SCS  International 
leadership,  and  SIMVAL  99  should 
consider  how  the  M&S  community  at  large 
can  be  better  informed  about  W&A 
issues  -  and  then  take  appropriate  action. 

i.  Do  distributed  simulation  and  parallel 
discrete  event  simulation  (DBS)  pose  even 
greater  challenges  for  M&S  W&A? 

•  Discussion: 

Yes.  Many  of  these  challenges  occur 
because  of  M&S  complexity  and  size, 
which  makes  comprehensive  testing  both 
difficult  as  well  as  time-consuming  and 
resource  intensive. 
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j.  Do  we  need  any  special  attention  to  V&V 
of  reused  objects  following  component- 
based  engineering  or  object  oriented 
paradigm  (OOP)? 

•  Discussion: 

Yes.  Just  because  a  component  passes  a 
V&V  test  suite/review  at  the  component 
level  does  not  ensure  that  component  will 
always  work  and  perform  correctly  when 
combined  with  other  components.  V&V 
must  always  be  addressed  as  a  multi¬ 
layered  process  with  iteration  as  needed  to 
ensure  credibility  and  suitability  of  both 
individual  components  and  their  combined 
use. 

4.  Synopsis  from  Subgroup  B  of  WGl 
a.  Phase-Specific  Verification  Processes 
Verification  typically  consists  of  a  set  of 
phase-specific  processes  including: 

•  Requirements  Verification 

•  CM  Verification 

•  Design  Verification 

•  Implementation  Verification 
Verification  depends  heavily  upon  and  is 
significantly  affected  by  the  type  of  model  and 
simulation  (M&S)  to  which  it  is  applied.  Because 
of  the  differences  in  application  and  the  vast 
differences  in  size,  complexity  and  fidelity  of 
various  M&S,  it  must  be  realized  that  none  of 
these  verification  processes  is  a  “one  size  fits  all” 
approach.  With  this  in  mind,  there  are  certain 
activities  that  are  always  performed  on  virtually 
every  V&V  program.  These  can  be  considered 
foundation  elements.  Then,  because  of  the 
differences  in  M&S  and  its  application,  there  are 
electives.  So,  much  like  a  curriculum,  there  are 
required  activities  and  elective  ones.  Moreover, 


even  in  the  required  or  foundation  elements  there 
will  be  variations  in  the  specific  tools  and 
techniques  used  for  problem  solving.  These 
decisions  are  based  on  corporate  culture,  sponsor 
preferences,  developer  engineering  environments, 
GOTS/COTS  tools  and  the  size  and  significance 
of  the  M&S  product  to  which  V&V  is  being 
applied. 

This  report  reminds  the  V&V  practitioner  to 
select  the  tools  and  techniques  based  on  some 
form  of  economic  and  technical  analysis.  Figure 
WG  1-1  contains  a  list  of  considerations  that 
should  assist  in  this  process  of  tool  and  technique 
selection.  Further,  this  report  does  not  endorse 
specific  tools  but  rather  uses  specific  ones  only  as 
examples  of  the  types  of  tools  under  discussion. 

In  general,  the  simpler  tools  often  tend  to  be 
adequate  for  V&V,  whereas  they  may  not  always 
satisfy  the  M&S  developer’s  needs.  Thus,  major 
objectives  of  tool  selection  should  be  to  acquire 
tools  that  are  easy  to  learn,  offer  the  specific 
capabilities  needed  and  are  as  economical  as 
possible  while  providing  the  necessary 
capabilities.  Tool  selectors  should  try  hard  not  to 
trade  essential  features  for  cost  savings,  as  this 
would  be  highly  counterproductive. 

One  of  the  recommendations  described  in  Figure 
WG  1-1  focuses  on  selection.  To  further  expound 
upon  this  idea,  the  V&V  planner  must  understand 
the  tasVs  that  need  to  be  accomplished  in  each 
phase.  From  this  list  of  essential  activities,  V&V 
must  determine  which  ones  need  a  formal  method 
or  procedure.  Those  needing  formal  methods  are 
examined  to  determine  whether  a  tool  or 
technique  is  preferred.  When  a  tool  is  indicated, 
perform  the  steps  in  Figure  WG  1-1. 
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Pick  Tools  that  Span  as  Many  Phases  as  Possible 
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Figure  WG  1>1.  Tool  Selection  Considerations 


b.  A  Generic  Tool  Profile 
Figure  WG  2-2  provides  a  high-level  view  of  the 
M&S  development  cycle  and  associated  W&A 
phases.  The  main  purpose  in  this  form  of 
representation  is  to  indicate: 

•  How  closely  aligned  V&V  is  with 
development  or  preparation  of  the  M&S. 

•  How  V&V  can  share  and  leverage  off 
development  tools,  products  and 
processes. 

•  That  V&V  needs  some  tools  and 
techniques  of  its  own  regardless  of  how 
rich  the  development  tool  set  is. 


There  is  also  some  possible  variation  in  where 
(which  phase)  a  particular  tool  or  technique  is 
used  because  of  variations  in  the  development 
cycle.  For  example,  in  legacy  M&S,  V&V  is 
likely  to  have  little  or  no  opportunity  to  leverage 
off  development  artifacts  and  tools  and,  therefore, 
may  be  required  to  “go  it  alone.”  In  such  cases, 
V&V  will  require  a  sufficient  set  of  tools  to 
support  all  of  the  essential  tasks,  with  selection 
based  on  the  goodness  and  maturity  of  the 
existing  product,  its  intended  use,  criticality  of  the 
application  and  various  other  factors. 
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1.  Analyze  each  phase  and  determine  those  things  that  should  be 
accomplished 


2.  Compare  capabilities  of  each  candidate  tool  to  this  list;  pick 
least  number  of  tools  that  offer  widest  set  of  capabilities. 

3.  Evaluate  cost  perseat,  training,  maintenance  cost  and  select 
tools  that  offer  best  value. 

4.  Determine  what  can  /  should  be  shared  with  developer  and  other 
participants. 

5.  Tools  that  are  too  difficult  to  learn,  have  a  long  input  cycle  time, 
or  conflict  with  deveiopment  should  be  avoided. 


Figure  WG 1-2  Typical  Tool  Profile 


Notice  also  that  the  M&S  developer  or  provider 
should  be  required  in  its  contract  to  share  such 
things  as  the  configuration  management  system 
and  records,  software  and  documentation 
libraries,  test  data  and  any  archived  data  and 
information  from  the  tools  used  to  build,  analyze 
and  evaluate  the  M&S  product.  It  may  also  be 
possible  in  some  cases  to  use  an  existing  Software 
Engineering  Environment  (SEE)  or  test  bed  that 
was  used  to  build  and/or  test  the  M&S.  Although 


this  is  far  from  a  complete  treatise  on  the  subject 
of  verification.  Figure  WG  1-2  helps  establish  a 
contextual  framework  for  the  specific  sections 
that  follow. 

Some  kind  of  organized  approach  to  W&A  costs 
is  required  if  meaningful  cost  estimation  for 
W&A  activities  is  to  be  viable.  Figure  WG  1-3 
illustrates  the  kind  of  factors  that  need  to  be 
considered  in  estimating  W&A  costs. 
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For  each  task: 


Cost  Estimation  Algorithm 


where: 
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For  all  tasks: 


where: 


Cost  Calculation: 
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=  Raw  counts  that  initially  populate  each  mode!  matrix 
K  Adjustment  factor  that  better  matches  model  to  application 
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=  Sc  Hoe© 

ici 

=  Total  level  of  effort  in  manhours  for  all  tasks  x  size  and  complexity 
=  Size  in  SLOC  x  complexity  in  function  points,  entities,  or  functional  elements 
=  Maximum  no.  of  tasks  included  in  model  matrix 


=  Tloe  X  Ac 

=  Average  cost  of  W&A  labor  per  manhour 
=  Cost  of  W&A  by  task,  phase,  and  project 


Figure  WG 1-3  Cost  Estimation 


c.  Issues  mid  Recommendations 
Requirements:  Verification  would  benefit  greatly 
from  formal  methods  and  better,  more 
comprehensive  CASE  tools;  however,  their  cost 
benefits  are  most  apparent  when  embraced  first  by 
the  developer  and  then  shared  by  V&V. 

CM  Verification  (and  Validation)  badly  needs  a 
unified,  standard  approach  to  defining  CMs. 

This,  in  turn,  spawns  the  need  for  a  tool  or  tools 
to  provide  this  service.  It  may  require  one 
approach  for  High  Level  Architecture  (HLA) 
based  M&S  that  use  the  CMMS  and  a  another  for 
those  M&S  that  do  not. 

New  development  methodologies  for  M&S  design 
and  coding  that  include  fourth  generation 
languages  (4GLs),  autocode  generators,  etc.  will 
require  new  adaptations  of  the  VV&A  process. 
These  approaches  will  typically  eliminate  most  of 
the  Implementation  Verification  activities. 

Current  CASE  tools  do  a  good  job  on  design  and 
code  and,  as  long  as  these  tools  are  shared  with 
V&V,  this  is  not  a  particularly  problematic  issue. 
V&V  should  be  discouraged  from  instituting 


elaborate  CASE  tools  as  part  of  the  V&V 
approach  that  are  different  from  those  used  in 
development. 

Procurement  organizations  need  to  be  aware  of 
the  needs  of  V&V  to  share  tools  and  data  from  the 
developer  and  include  these  requirements  in  the 
RFPs  and  SOWs  for  the  development  contracts  to 
ensure  adequate  support  for  V&V. 

The  opinion  of  the  group  was  that  W&A  needs 
better  propagation  throughout  the  M&S 
community. 

Recommend  that  our  university  system  and 
continuing  education  curricula  be  encouraged  to 
increase  emphasis  on  V&V  and  M&S  tool 
technology. 

Increase  flow  from  projects  into  M&S 
Repositories  for  case  studies,  tools,  methods,  and 
cost  data. 

5.  Findings  and  Observations:  Requirements 
Verification:  Current  State 

a.  There  is  a  need  to  establish  a  better 
understanding  of  the  problem  by  all 
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participants  —  customer,  user,  developer 
and  W&A  practitioners  —  at  the  beginning 
of  the  M&S  effort 

b.  There  is  a  lack  of  analytical  formalism  and 
accompanying  methodology  in  this  phase 

c.  Tool  selection  should  be  based  on  cost 
versus  tool  compatibility  and  availability  as 
well  as  its  ability  to  match  the  application 

d.  There  is  a  great  need  for  training  and 
education  as  well  as  specific  talent  and 
acquired  skills 

e.  Tools  need  to  be  extensible  and  scalable  to  a 
wide  range  of  W&A  efforts 

f.  Data  needs  to  be  persistent,  maintainable 
and  easy  to  change  and  verify 

g.  The  impact  of  large,  multi-user/multi-site 
projects  needs  to  be  considered  in  the 
W&A  methodology  and  tool  selection 
process. 

6.  Requirements  Verification:  Future  Needs 

a.  Better,  cheaper  requirements  analysis  tools 

b.  Improvements  in  requirements  management 

c.  Improved  capabilities  to  support  analytical 
studies  especially  with  tools 

d.  More  complete  integration  with  downstream 
phases  and  tools 

e.  Enhanced  formal  processes  and  methods, 
tools  and  formal  specification  languages 

f.  Repository  of  requirements  characteristics  — 
persistent,  traceable,  and  reusable. 

7.  CM  Verification:  Current  State 

a.  Tool  selection  should  be  based  on  cost 
versus  tool  compatibility  and  availability  as 
well  as  its  ability  to  match  the  application 
and  span  between  requirements  and  design 
phases 

b.  Most  CMs  are  not  clearly  understandable  by 
all  evaluators 

c.  Should  the  “use  case”  analysis  from  Unified 
Modeling  Language  (UML)  be  applicable  to 
CMs? 

d.  Bi-directional  tracing  between  CM  and 
object  model  is  required 

e.  Lack  of  a  widely  accepted  view  of  the  CM 
reduces  its  utility 

f.  Lack  a  Conceptual  Modeling  tool  that  can 
and  should  be  used  throughout  the  rest  of 


the  M&S  development  and  W&A 
processes. 

8.  CM  Verification:  Future  Needs 

a.  Need  to  continued  development  and  V&V 
of  the  CMs  of  the  Mission  Space  (CMMS) 
and  other  authoritative  sources 

b.  Need  standard  methodologies  and  tools  to 
build  CMs 

c.  Enhancement  of  CMMS  and  other  data 
sources  and  extraction  tool  sets  that  help 
automate  and  assist  the  building  of  the  CM 

d.  Consistent  practices  for  inputs  to  CMs 

e.  Automated  traceability  from  requirements  to 
the  CM  and  from  the  CM  to  design. 

9.  Design  Verification:  Current  State 

a.  Although  they  sometimes  do,  V&V  tools 
should  not  compete  unnecessarily  with 
developer’s  tools  (acquire  necessary  copies 
for  V&V) 

b.  Lack  of  comprehensive  tools  to  examine  and 
evaluate: 

c.  Key  algorithms,  behaviors,  performances 
(e.g.,  states,  functions) 

d.  Data  and  databases 

e.  Interfaces  (both  internal  and  external) 

10.  Design  Verification:  Future  Needs 

a.  Repository  of  data  and  information 
describing  design  errors  by  type  and  class 

b.  Automated  testing  aids  associated  with 
design  that  would  become  part  of  this 
repository 

c.  More  effective  tools  to  analyze  algorithms, 
data  and  interfaces  as  well  as  control  and 
data  flow  or  that  represent  the  object 
oriented  views  of  the  design  in  easy  to 
understand  notation 

d.  Visualization  tools  to  examine  consistency 
across  different  design  methodologies 
within  a  single  end  product. 
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11.  Implementation  Verification:  Current 
State 

a.  Difficulty  in  defining  how  to  select  and 
ensure  adherence  to  coding  standards  and 
good  practices 

b.  Highly  dependent  on  subject  matter 
knowledge  (programmer) 

c.  Need  to  improve  process  of  selection  of 
appropriate  metrics  for  specific  types  of 
problems 

d.  Need  to  improve  the  use  and  effectiveness 
of  support  tools,  which  is  heavily  dependent 
on  developer’s  software  engineering 
environment  (SEE). 


12.  Implementation  Verification:  Future 
Needs 

a.  Use  of  new,  more  productive  and  easier  to 
analyze  languages  for  modeling  and 
simulation 

b.  Static  and  dynamic  code  analyzers  for  M&S 
languages  that  currently  have  none 

c.  Better  maintenance  and  easier  reuse  of 
legacy  models  and  analytical  tools 

d.  Appropriate  application  of  talent  and 
emerging  technologies 
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The  synopsis  begins  by  identifying  the  briefings  validation  methodology  and  technology  are 
presented  to  Working  Group  2.  Then  there  is  a  presented, 

brief  overview  of  how  WG  2  functioned.  Then 
findings  /observations  about  current  and  future 
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1.  WG  2  Briefings: 

a.  “Modular  Semi-Automated  Forces 
(ModSAF)  Experimentation  (VV&A, 
Physical  Models,  Data)”  by  John  Thomas 

b.  “A  Construct  for  the  Use  of  SMEs”  by  Scott 
Harmon 

2.  WG  2  Overview: 

a.  The  TOR  provided  guidance  for  WG  2 
deliberations.  This  guidance  focused  the 
discussions  on  the  following  areas. 

•  Subject  Matter  Expert  (SME)  selection, 
orientation,  management  and  use. 


especially  in  validation  of  CMs,  and 
simulation  outputs. 

•  Fidelity 

•  Correlation  of  representational  fidelity 
with  application  needs 

•  Criteria  for  establishing  needed  levels  of 
simulation  fidelity 

•  Technology  (e.g.  automation)  potential  to 
support  validation  methodology 

•  Simulation  CMs  -  correlation  among  the 
various  levels. 
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Figure  WG  2-1  M&S  Development  Cycle  and  V&V 


WG  2  organized  its  discussions  according  to  the 
various  phases  and  validation  tasks  that  parallel 
the  development  cycle.  Figure  WG  2-1  depicts 
the  M&S  development  cycle  described  in  the 
RPG.  The  four  validation  tasks  that  include  or 
affect  validation  tasks  are  described  below. 

•  Developing  validation  criteria  -  provide 
standards  for  comparing  models  with  the 
real  world. 

•  Validating  the  CM  -  determine  the  degree 
to  which  the  CM  is  an  accurate 
representation  of  the  real  world  from  the 
perspective  of  the  intended  uses  of  the 
model. 

•  Validating  the  input  data  -  determine  that 
the  data  for  use  in  M&S  is  suitable  and 
appropriate  for  the  model's  intended  use. 

•  Validating  the  application  -  assess  that  the 
developed  M&S  outputs  compared  within 
specified  error  with  what  is  known  about 
the  real  world  using  the  validation  criteria 

The  discussion  concerning  validation  criteria  led 
to  the  identification  of  several  elements  that  were 
needed  to  adequately  specify  validation  criteria. 
These  are: 

•  Assumptions  and  limitations  related  to  the 
problem  or  design  of  experiment 


•  Required  Accuracy 

•  Required  Scalability 

•  Level  of  confidence  required  in  validation 
results 

•  A  means  of  determining  what  types  of 
validation  are  appropriate  for  this  problem 

•  An  issues-to-measures-to-data  dendritic 
The  discussion  then  focused  on  what  tools  were 
or  would  be  useful  to  help  one  identify  and 
present  a  full  description  of  the  validation 
criteria. 

The  discussion  concerning  CM  validation  rapidly 
turned  into  a  discussion  of  just  what  constituted  a 
CM.  The  group  agreed  upon  using  the  definition 
that  is  sununarized  in  Chapter  1  of  the  current 
DoD  W&A  RPG.  This  summary  states  that  the 
following  elements  constitute  a  CM: 

•  Description  of  Requirements  -  the 
modeling  pieces 

•  How  the  pieces  are  planned  to  interact 

•  How  they  will  work  together  (integration 
approach) 

•  Descriptions  of  equations  &  algorithms  to 
be  used 


27 


•  Assumptions  &  limitations  growing  out  of 
selected  equations  &  algorithms 

•  How  these  assumptions  and  limitations  are 
expected  to  impact  on  the  simulation's 
ability  to  meet  the  problem  requirements 

The  group  concurred  that  CMs  should  be 
described  with  this  level  of  specificity. 

Other  issues  related  to  CM  validation  that  were 
discussed  included  the  types  of  SMEs  and  what 
qualifications  they  should  have;  what  questions 
are  appropriate  to  ask  in  SME  reviews,  what 
evaluation  criteria  should  be  used  and  what 
knowledge  acquisition  tools  are  appropriate.  The 
topic  of  SME  selection  was  pursued  in  some 
depth  and  included  an  abstract  presentation  that 
related  SME  qualifications  to  the  nature  of  the 
problem. 

SMEs  are  an  essential  part  of  any  simulation 
validation  effort.  However,  several  constraints 
exist  in  order  to  maximize  the  value  of  their 
contributions  that  a  developer  must  take  into 
account  to  optimize  the  use  of  SMEs.  The 
developer  has  a  model  needing  validation  (M)  and 
the  specified  purpose  intended  for  the  model  (P). 
From  these,  the  SMEs  must  build  their  own 
internal  models  of  the  model  or  simulation  and  its 
purpose.  This  requires  the  SMEs  to  be  able  to 
perceive  the  model  and  purpose  (ideally  with 
minimum  distortion).  This  process  is  seldom 
perfect  —  especially  for  complex  models,  for 
models  with  unfamiliar  representations  (e.g., 
neural  networks)  and  for  models  with  involved 
purposes  or  those  unfamiliar  to  the  SME.  The 
developer  must  therefore  take  extra  precautions  to 
reduce  the  probabilities  of  distortion  occurring 
during  these  transfers.  After  the  SMEs  have 
developed  their  internal  models  of  the  model 
under  validation  and  its  purpose,  they  must  put 
those  internal  models  into  a  form  that  they  can 
compare  with  their  knowledge  of  the  domains  of 
their  expertise.  This  is  essentially  a  normalization 
step.  The  SMEs  then  compare  these  normalized 
models  with  their  expert  knowledge  to  assess 
their  validity  for  the  intended  purpose.  Each  of 
the  arrows  in  this  illustration  represents  an 
opportunity  for  distortion  that  introduces  error 
into  the  validation  process.  Of  course,  any  such 
error  contributes  to  validation  errors.  The 
developer  or  validation  agent  or  both  must 


recognize  these  error  sources  and  choose  SMEs 
and  representations  to  minimize  their  effects. 

This  suggests  several  criteria  for  selecting  SMEs 
to  validate  a  particular  model  for  a  specific 
purpose.  Obviously,  the  SMEs  must  possess  the 
domain  knowledge  that  relates  to  both  the  model 
under  validation  and  to  the  purpose  for  which  it  is 
being  validated.  However,  beyond  this  well 
understood  requirement,  several  more  criteria 
exist  to  reduce  the  possibilities  of  introducing 
errors  into  the  validation  process.  In  other  words, 
simply  choosing  SMEs  with  the  necessary  domain 
knowledge  is  not  sufficient  to  guarantee 
satisfactory  validation.  SMEs  must  be  able  to 
correctly  perceive  both  model  and  purpose  to 
form  accurate  internal  models  that  they  will  then 
compare  with  their  domain  knowledge. 

Inaccurate  perceptions  of  model  and  purpose  will 
lead  to  inaccurate  validation  results  as  surely  as 
inappropriate  domain  knowledge.  Developers 
must  ensure  that  the  SMEs  understand  the 
representations  of  these  elements.  These 
problems  are  often  seen  when  models  use 
representations  that  are  unfamiliar  to  SMEs  (e.g., 
production  rules).  SMEs  must  also  be  able  to 
normalize  their  perceptions  of  the  model  under 
validation  and  its  purpose  and  compare  those 
perceptions  with  their  domain  knowledge.  These 
steps  are  most  often  ignored  by  developers  and 
validation  agents  who  operate  under  the 
assumption  that  if  the  SMEs  have  the  required 
domain  knowledge  then  they  can  perform  the 
required  validation.  Normalization  and 
comparison  errors  are  difficult  to  detect  and 
probably  result  in  far  more  validation  errors  than 
currently  appreciated.  Finally,  the  SMEs  must  be 
able  to  clearly  communicate  their  assessments. 
This  source  of  error  can  be  minor  in  most  cases 
with  the  proper  SME  selection.  Many 
applications  require  multiple  SMEs  since  no 
single  person’s  domain  expertise  covers  all  of  a 
model  or  its  purpose.  In  these  cases,  the  arrows  in 
the  previous  slide  represent  communications 
between  people,  always  sources  of  error  and 
distortion. 

In  other  more  practical  but  less  robust  terms,  a 
suggested  list  of  the  different  types  of  expertise 
needed  for  a  CM  review  include: 

•  Operational 
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•  Analytic  (possessing  an  understanding  of 
the  problem) 

•  M&S  being  considered 

•  Technology  of  the  system(s)  being 
modeled 

•  Modeling  Techniques  -  Application 
Specific 

The  data  validation  discussion  led  to  a  list  of 
questions  about  the  data  that  are  needed  to 
adequately  assess  its  quality.  The  questions  that 
were  identified  are  listed  here. 

•  What  is  the  source?  (Data  and 
Algorithms) 

•  Is  it  authoritative? 

•  What  algorithms  are  implied  or  embodied 
in  the  data? 

•  Is  it  valid  for  intended  use? 

•  How  good  does  it  need  to  be? 

•  What  is  the  type,  compatibility,  format  and 
accuracy  of  the  data? 

•  What  is  the  classification,  cost  and 
availability  of  the  data? 

•  Are  the  data  proprietary? 

•  What  is  the  data  pedigree? 

The  discussion  of  application  validation  began 
with  a  definition  or  explanation  of  what  is  meant 
by  application  validation.  The  WG  agreed  that  it 
is  a  comparison  of  the  model  outputs  to  the  real 
world  and  what  we  know  about  it  and  making  a 
determination  about  its  acceptability.  This  can  be 
done  using  a  number  of  techniques.  The  two 
principle  approaches  are  quantitative  and 
qualitative.  The  quantitative  approach  can 
employ  statistical  methods  and  usually  is  done 
either  through  results  validation  or  benchmarking. 
Qualitative  approach  is  usually  done  through  face 
validation.  The  process  recognized  by  the  WG 
follows  three  basic  steps:  generate  M&S 
predictions,  collect  retd  world  data  and  compare 
the  two  methods  to  determine  the  best  approach. 

These  discussions  culmination  in  discussions 
about  what  tools  and  techniques  were  typically 
used  in  the  various  steps,  what  deficiencies  were 
perceived  and  what  improvements  or 
advancements  are  needed  to  make  the  validation 
process  more  effective. 

In  pursuing  these  discussions,  the  WG  agreed  that 
the  cost  of  validation  is  greatly  influenced  by  the 
validation  techniques  selected,  the  tools 
employed,  and  especially  the  initial  validation 


requirements.  The  WG  recognized  that  the  most 
important  factors  that  affect  validation  costs  are: 

•  The  determination  of  what  types  of 
validation  are  appropriate  for  a  given 
problem  or  model  development  effort 

•  The  determination  of  what  level  of 
confidence  is  needed  in  the  model  results 
so  that  the  depth  and  scope  of  validation 
can  be  properly  planned. 

•  The  cost  of  obtaining  real  world  data 
(from  tests  or  operations)  for  use  in 
validation. 

3.  Findings  and  Observations 
a.  The  State  of  Validation  Methodology  is 
Marginal  and  The  State  of  the 
Technology  is  Poor.  The  need  to  Verify 
and  Validate  (V &V)  M&S  has  been  a 
requirement  since  the  first  model  and 
simulation  was  written  and  has  received 
increased  emphasis  with  the  past  ten  years. 
However,  despite  this  emphasis  and  while 
the  required  validation  processes  are 
generally  well  known,  they  are  not 
rigorously  followed  nor  are  integrated 
together  today.  Further,  while  it  is  fairly 
easy  to  derive  many  properties  required  for 
desirable  tools,  it  is  difficult  to  find  tools 
that  incorporate  these  properties.  Some  of 
the  properties  required  are  more  difficult  to 
derive.  These  would  include  selecting  the 
right  real  world  variables  to  be  mapped  in 
the  CM.  In  general,  the  current  state  of 
validation  is  brute  force  labor.  Using  the 
Validation  tasks  identified  above  as 
organizing  elements,  a  more  specific 
discussion  of  these  assessed  states  is 
discussed  below. 

•  Develop  Model  Validation  Criteria.  There 
are  many  existing  tools  to  support  the 
methods  described  above.  They  include 
word  processors,  spreadsheets,  database 
management  systems,  system  engineering 
tools,  subject  matter  experts  (SME), 

Model  and  Simulation  Requirements 
Repository  (MSRR),  statistical  and 
simulation  tools  and  knowledge 
acquisition  and  engineering  tools. 
However,  improvements  in  the  tools  were 
seen  to  be  required.  Knowledge 
acquisition  and  engineering  was  noted  as 
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offering  a  substantial  challenge  for 
improvement. 

•  Validate  CM.  Although  some  tools  exist 
to  support  CM  Validation,  they  were 
assessed  to  be  particularly  weak.  Current 
tools  include  Computer  Assisted  System 
Engineering  Tools  (CASE)  tools  such  as 
Paradigm  Plus,  Cadre  TeamWork  and 
Design/IDEF;  "Dynamic"  CASE  tools  for 
meta-simulations  such  as  Design/CPM, 
ModSIM,  UHINK,  flowcharting  and 
structured  walk-throughs. 

•  Validate  Data.  Current  tools  include 
database  management  systems  to  support 
the  traceability  of  data  and  authoritative 
data  sources,  data  quality  engineering  to 
assess  the  range  williin  which  the  data  is 
valid  and  any  logical  errors  in  the  use  of 
the  data,  and  data  modeling  tools  to  show 
the  relationship  between  data.  Tools  in 
this  area  were  assessed  to  be  mature  for 
software  engineering  purposes,  but  weak 
for  M&S  requirements  definition  and 
management  purposes. 

•  Validate  Application.  Current  tools  to 
accomplish  this  are  numerous  statistical 
analysis  tools,  visualization  in  two  or  more 
dimensions  (MUSE  technology),  the  used 
of  After  Action  Reviews  (AAR)  that  are 
interactive  and  provide  model  results  in  a 
maimer  understandable  to  the  users,  and 
SEDRIS-like  tools.  It  was  the  consensus 
of  the  WG  that  current  tools  need  to  be 
significantly  improved. 

4.  Near  Term  Findings  and  Observations  for 

Validation  Methodology 

•  Improve  The  Description  Of  Validation 
Methodology  In  The /?PG.  TheWG 
concluded  that  while  validation 
methodology  is  described  RPG  for  general 
reference  purposes  that  the  description 
was  incomplete  for  task  planning  and 
execution. 

•  Establish  methodology  for  independent 
peer  review  of  CMs.  The  WG  recalled 
from  previous  meetings  on  validation  that 
a  promising  approach  for  validating  the 
CM  was  in  having  it  reviewed  by  persons 
independent  of  the  current  M&S 
development  effort  who  had  built  CMs  for 


other  M&S  projects.  One  individual  from 
the  WG  cited  his  own  experience  as 
evidence  that  this  was  a  sound  practice. 

•  Analyze  Current  Validation  Methods 
Towards  Integrating  Them  Into  A  More 
Coherent  Methodology.  The  WG  assessed 
that  individual  validation  processes  were 
poorly  integrated  between  themselves  and 
that  this  fact  mattered  to  the  overall 
validation  of  the  M&S  being  developed. 

•  Incorporate  Continuous  Process 
Improvements  Into  M&S  Development. 
The  WG  observed  that  M&S  development 
is  largely  an  iterative  process  that  cycles 
repeatedly  from  requirements  analysis 
through  applications.  This  cyclic  effect 
begs  for  continuous  process  improvement 
in  the  sense  of  total  quality  management. 

•  Develop  Guidance  For  The  User  And 
SMEs  In  M&S  Development.  The  WG 
developed  a  model  for  SME  involvement 
in  the  validation  model  that  can  be  used  in 
developing  mles  for  SME  involvement. 
Also,  the  Conceptual  Model  of  the 
Mission  Space  (CMMS)  and  CM  activities 
were  identified  as  important  to  user 
involvement.  Finally,  the  WG  developed  a 
generic  set  of  model  validation  criteria  that 
were  linked  to  the  intended  use  of  the 
M&S  that  were  determined  to  be  an 
important  mechanism  for  measuring 
validation  success. 

•  Expand  Methodology  To  Include 
Simulation  Based  Acquisition  (SBA)  And 
Joint  Experimentation  (JE)  Requirements. 
SBA  and  JE  are  DoD  concepts  that 
envision  new  roles  for  M&S  use.  The  WG 
concluded  that  it  was  important  for  DoD  to 
investigate  the  implications  of  these 
concepts  to  the  vgdidation  of  M&S  for 
these  purposes.  Also  related  to  this 
finding ,  is  the  observation  that  the 
validation  process  in  the  distributed 
simulation  environment  is  still  not  fully 
understood  although  efforts  have  been 
made  previously  to  do  so. 

•  Develop  Tools  Or  Tool  Sets  Having  The 
Identified  Required  Properties  For 
Validation.  (E.G.  Develop  Universal  Tool 
Set  To  For  Conceptual  Modeling.  Suggest 
starting  a  development  of  a  CM  template). 
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While  the  RPG  has  a  fairly  good  definition 
and  description  of  the  CM  and  there  are 
many  known  useful  tools  that  could  be 
used  to  assist  in  its  description,  the  WG 
decided  that  there  is  insufficient  guidance 
to  determine  when  a  sufficiently  complete 
CM  has  been  achieved  to  specify  a  design. 
Further,  it  is  unknown  what  an  objective 
description  of  a  tool  or  tool  set  is 
necessary  to  achieve  the  objective  CM. 
This  difficulty  is  complicated  by  the 
iterative  nature  of  CM  development.  Even 
after  design  or  implementation  is 
underway,  the  development  process  may 
revisit  and  revise  the  CM.  However,  while 
the  CM  may  always  be  incomplete,  it  is 
believed  that  an  objective  CM  can  be 
defined  for  each  M&S. 

•  Leverage  instrumentation  technology  to 
improve  the  quality  and  scope  of  real  data. 
Experience  from  STOW,  ModSAF  and 
A2ATD  revealed  the  value  of 
instrumenting  M&S  to  generate  data  for 
configuration  management,  statistical 
diagnostics  and  AAR.  The  WG  believes 
that  instrumentation  of  M&S  can  be  a 
powerful  tool  for  validation. 

•  Conduct  education  in  the  use  of 
technology  to  be  compatible  with  the 
technology.  The  WG  decided  that  the 
introduction  of  new  methods  and 
technology  into  M&S  validation  would  be 
placed  at  a  disadvantage  if  there  were  not 
parallel  efforts  to  educate  users  and 
developers  in  their  implementation.  For 
example,  the  incorporation  of  new 
capabilities  to  provide  statistical  analysis 
such  as  for  defining  levels  of  confidence 
or  risk  would  not  be  useful  unless  the 
users  to  relate  these  to  meaningful 
measures. 

5.  Far  Term  Findings  and  Observations  for 

Validation  Methodology.  Invest  in  research  for 

the  following  areas; 

•  Modeling  Theory  To  Better  Understand 
The  Relationship  Between  That  Of  The 
Real  World  And  The  Formal  Systems  That 
Represent  Them.  Validation  requires 
linking  model  representations  with  the  real 
world.  Specifically,  the  CM  is  an 


abstraction  of  the  real  world.  How  one 
chooses  the  finite  subset  of  the  real  world 
and  maps  it  into  the  CM  is  difficult.  For 
example,  there  are  no  good  rules  for 
selecting  what  variables  should  be  inputs 
or  parameters  in  a  model  having  great 
impact  on  the  quality  of  validation.  While 
it  was  observed  that  tools  seldom  are  a 
replacement  for  talent,  it  was  also  noted 
that  good  theory  is  a  talent  enhancer. 

•  Collection  of  Reliable,  Reproducible 
Experimental  Data.  Relating  to  the  need 
for  research  into  modeling  theory  for 
M&S  development  (and  validation)  is  the 
need  to  conduct  field  tests  in  order  to  test 
the  theory  and  to  provide  an  analytical 
basis  to  validation  testing. 

6.  Far  Term  Findings  and  Observations  for 

Validation  Technology.  Invest  in  research  for 

the  following  areas: 

•  The  Measurement  Of  Human  Performance 
(Both  Human-bi-The-Loop  (HTTL)  And 
Human  Representations).  Human 
behavior  replicated  within  M&S  and 
interacting  with  M&S  is  problematic  but 
also  an  important  part  of  the  validation 
problem.  This  aspect  of  M&S  will 
become  increasingly  important  in  SBA 
and  JE. 

•  Knowledge  Acquisition  And  Engineering 
Tools.  Requirements  include 
development,  analysis  and 
conceptualization  demand  knowledge 
acquisition  and  engineering  in  order  to 
transition  the  semantics  of  the  real  world 
into  the  syntax  of  a  CM  and  its  objective 
M&S.  Currently,  this  is  a  labor  intensive 
effort  that  provides  great  opportunities  for 
technology  to  improve. 

•  Complexity  (Modeling  Information  Based 
Phenomena).  The  WG  observed  that  the 
state  of  the  art  in  validating  some  real 
world  phenomena  varied  between  the 
phenomena  being  modeled.  For  example, 
many  physics  based  phenomena  such  as 
ballistics  are  modeled  in  analytic 
expressions  using  assumptions  that 
provide  a  high  degree  of  confidence  in 
their  use.  However,  examples  involving 
visualization  such  as  Uuget  acquisition. 
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interactions  of  battlefield  entities  and  better  understand  how  they  can  be 

other  information-based  phenomena  modeled 

illustrate  the  M&S  community's  need  to 
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1.  WG  3  Briefings 

The  following  briefings  were  presented  in 
WG3. 

a.  Costing  Of  Model  And  Simulation 
(M&S)  Verification,  Validation,  And 
Accreditation  (W&A)  Within  DoD  — 
A  Managed  (Marginal)  Investment 
Strategy  by  Robert  Gravitz  and  Bill 
Waite 

b.  Overview  of  Six  Major  W&A 
Processes  and  Associated  Cost  Models 
by  Robert  O.  Lewis  to  a  combined 
session  for  WGs  1  (Verification 
Technology)  and  3  (W&A  Costs) 

c.  Advanced  Distributed  Simulation 
(ADS)  Work  Breakdown  Structure 


(WBS)  AS  of  13  Jan  1999  by  LTC 
Cannon 

2.  Introduction 

Within  M&S  economics,  in  particular 
W&A,  there  are  both  cost  and  benefit 
components  as  illustrated  in  Figure  WG  3-1 
We  concentrated  on  the  cost  components. 
We  identified  M&S  technologies  and 
assessed  their  impact  on  W&A  costs. 

Then  we  cross-walked  the  technologies  and 
cost  to  factors  that  either  reduced  or 
compounded  V&V  costs. 
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Figure  WG  3-1  W&  A  Interactions 


3.  Cost  Factors  Descriptions 

We  identified  the  following  cost  factors  as 

influencing  V&V  costs: 

a.  Stability  of  Requirements.  Variability  of 
Simulation  Requirements  as  a 
consequence  of  the  inherent  instability 
of  the  problem  or  variability  o  user 
expressed  needs/  Requirements 

b.  Complexity  of  User  Requirements. 
Number  /complexity  of  Simulation 
Requirements  as  a  consequence  of  the 
inherent  difficulty  of  the  problem 

c.  Security  Requirements.  Classification 
and  compartmentalization  of 
information  and  security  required  to 
access  that  information  (firewalls,  etc.) 

d.  Time  Available.  Schedule  (calendar) 
driven  program 

e.  M&S  System  Documentation.  Degree 
to  which  the  system  is  well  described  in 
appropriate  form  (documents,  etc.) 

f.  Required  Credibility.  The  degree  of 
credibility  of  results  or  the  stringency  of 
level  of  proof  required  by  authority 

g.  Probability  of  risk  x  impact  of  risk.  Net 
risk 

h.  Size/complexity  of  Unit  Under  Test. 
Size/complexity  of  the  Simulation 
system  and  /or  components 

i.  Maturity  of  The  Simulation 
Development  Process.  Akin  to  SEI 
CMM  but  correlated  to  simulation 
development 

j.  Availability  of  Reference  Data. 
Availability  of  data  used  to  compare 


simulation  results  to  (e.g.  validation 
reference  data) 

k.  COTS/GOTS  (make  or  buy  decisions). 
Visibility  (or  lack  thereof)  into  COTS 
(for  example)  and  its  inherent  quality 
(or  lack  thereof)  may  impact  the  cost  of 
VV&A;  need  to  develop  your  own 
(GOTS)  will  impact  cost  of  W&A 

l.  Competence  Of  V & V  Execution 
Agents.  Skill  and  ability  to  execute 
V&V  tasks  efficiently  and  effectively 

m.  System  Engineering  Process.  Quality 
and  maturity  of  entire  program 
management  processes  of  the  reference 
system  (above  Simulation  development 
process) 

n.  Consistency  Of  Enterprise  Culture. 
Degree  of  shared  appreciation  of  the 
mission  space,  simulation  system, 
architecture,  essential  business  and 
technical  W&A  process 

4.  Technologies  Area  Descriptions 
We  assessed  the  potential  for  the  following 
technologies  to  compound  or  reduce  V&V 
costs. 

a.  High  Order  Languages.  Programming 
and  simulation  languages  that  provide 
a  high  level  of  functionality  in  a 
concise  form. 

b.  Integrated  Development 
Environments.  Software  tools  and 
methods  that  help  organize  and 
automate  the  development  process. 

c.  Telecom  Infrastructure. 
Telecommunications  facilities  and 
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c.  Telecom  Infrastructure. 
Telecommunications  facilities  and 
architectures  available  to  support 
M&S  development/use. 

d.  Computational  Capability.  Speed, 
capacity  and  architecture  of  computing 
hardware. 

e.  Database  Technologies.  Tools  and 
methods  of  acquiring,  storing  and 
retrieving  data/information. 

f.  Advanced  Alternative 
Representations.  Advanced  methods 
including  for  example,  neural 
networks,  fuzzy  logic,  knowledge  and 
rule  based  systems,  genetic 
programming. 

g.  Program  Management  Tools/Business 
Practices.  Methods  and  tools  for 
planning,  scheduhng,  control, 
budgeting  and  other  tasks  for  program 
managers. 

h.  Automated  Repositories.  Large  scale 
distributed  databases  accessible  to  a 
wide  user  community,  for  example 
MSRR. 

i.  Knowledge  Discovery  in  Databases. 
Automated  knowledge  tools  for 
research  of  database  information  and 
recovery  of  specific  data  and  the 
mining  of  data  based  on  AI 
techniques. 

j.  Analysis  and  Visualization 
Technologies.  Methods  and  tools  for 
visualizing  and  analyzing  attributes 
and  relationships  of  the  real  world,  the 
simulation  system  and  related  data. 

k.  Synthetic  Environments.  Virtual 
environments  that  replicate  and  model, 
and  permit  immersion  into,  relevant 
operational  context  (includes  some. 


but  possibly  not  ah  of  cognitive 
performance  modulators  —  see 
SIMTECH  2007). 

l.  CM  Forms/Notations.  Practices, 
representational  schema  and  formal 
methods  that  facilitate  precise 
specification  of  models. 

m.  Operational  Practice.  Well-defined 
elements  of  technical  and  business 
practice  of  simulation  and  W&A  that 
facilitate  efficient  practice.  Level  of 
craft  skills  in  W&A. 

n.  Decision  Support  Tools.  Aids  to 
managers  and  analysts  for  identifying 
and  analyzing  options. 

o.  Security/Encryption.  Methods  and 
technology  for  protecting  systems  and 
data. 

p.  Standards.  Uniform  practices  and 
standards,  including  operational, 
system,  and  technical. 

q.  Test  Generation  Tools.  Methods  and 
tools  for  automated  generation  of  test 
cases. 

r.  Code  Generation  tools.  Methods  and 
tools  for  automated  generation  of 
source  code. 

s.  Scalability.  Tools  and  methods  to 
change  the  resolution  in  a  consistent 
manner;  to  change  the  breadth  and/or 
depth  of  simulation  with  consistency. 
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Table  WG  3-1  Cost  Factor  -  Technology 

Table  WG  3-1  summarizes  the  impact  of  costs.  Additional  considerations  are  summarized 

technologies  based  on  their  influence  of  V&V  in  Table  WG  3-2. 
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Table  WG  3-2  Additional  Considerations 


5.  Key  Points  and  Lessons  Learned: 

From  a  strategy  point  of  view,  the  greatest 
potential  cost-saving  technology  areas  are: 

a.  Program  management  tools  and 
technologies  appear  to  have  a  positive 
impact  on  W&A  across  the  board. 

b.  Automated  repositories,  to  make  existing 
W&A  data  is  available  in  all  areas  and 
historical  V&V  data.  This  helps  to  reduce 
the  research  required.  For  example,  if  you 
have  a  good  articulation  of  a  model’s 
previous  usage,  you  may  not  need  to  do  as 
much  V&V  as  might  be  required  if  little  is 
known  about  previous  usage.  There  are  also 
other  implications  of  this  item. 

c.  Database  technologies  allow  for  sharing  of 
data,  provides  consistency  of  form,  format 
for  data  and  help  with  documentation. 


d.  Knowledge  discovery  in  a  database  is 
almost  uniformly  positive,  but  may  result  in 
a  need  for  greater  security  because  it  allows 
the  user  to  aggregate  data,  possibly  violating 
security  requirements. 

e.  Standards  have  a  positive  intact  on  almost 
all  cost  factors  by  helping  to  provide 
consistency,  repeatability,  conforarity  and 
aid  collaboration. 

From  a  strategy  point  of  view,  the  technology 
areas  that  could  benefit  W&A  but  also 
potentially  have  a  negative  impact  on  cost  are: 

•  Advanced  alternative  representations  have 
both  a  positive  and  negative  impact  on  the 
cost  of  W&A  because  they  may  be  more 
complicated  to  understand  and  thus,  more 
difficult  to  articulate  credibility.  By 
providing  a  high  level  of  functionality. 
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these  methods  may  reduce  the  size  and 
complexity  of  modeling  a  unit  under 
testing. 

•  Security  technology  can  make  it  easier  to 
distribute  and  share  information.  If  done 
well,  with  the  ideas  of  being  able  to  share 
infomaation,  it  may  reduce  cost.  Security 
and  encryption  technology  may  limit 
accessibility  to  data,  thereby  increasing  the 
costofV&V. 

•  Scalability  may  make  it  easier  to  validate 
an  aggregate  model  if  component  models 
are  validated.  Scalability  may  result  in  a 
more  complex  model,  with  the 
consequence  that  the  people  who  do  the 
V&V  may  need  to  be  more  competent  in 
the  spectrum  of  what  is  being  modeled 


6.  Issues  and  Concerns 

a.  The  expression  of  cost  factors, 
dependencies  and  relationships  isn’t  well 
defined  for  M&S  nor,  consequently,  for 
W&A. 

b.  The  allocation  cost  to  W&A  versus 
allocation  to  system  development  (as  well  as 
product  quality  and  W&A  costs)  need  to 
be  better  defined  and  appreciated. 

c.  Our  evaluation  and  judgment  of  the 
significance  of  technology  upon  the 
economy  W&A  is  rough  but  suggestive. 

The  most  effective  way  to  address  these  concerns 
would  be  the  establishment  of  an  extensive 
database  of  W&A  costs  for  a  large  collection  of 
M&S  programs.  Analysis  of  the  database  would 
help  define  more  effective  cost  measures  and 
allow  for  tracking  productivity. 
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Security/encription 

- 

- 

- 

+ 

+- 

+- 

Table  WG  3-3  Cost  Factor  Technology  Crosswalk 


7.  Cost  Factor  —  Technology  Crosswalk: 
Schedule  time  availability  is  the  factor  that 
appears  to  have  the  greatest  potential  for 
improvement  through  technology. 

In:^>rovements  in  technology  have  a  significant 
potential  for  reducing  cost  by  increasing  the 
availability  of  reference  data.  This  helps  to 
reduce  the  amount  of  re-work  needed,  and  helps 
to  reduce  the  reliance  on  subject  matter  experts 
for  validation. 

Technology  has  potential  for  enhancing  the 
expertise  available  to  V&V  agents. 

All  process-related  cost  factors  appear  to  be 
strong  candidates  for  application  of  technology, 
resulting  in  a  decrease  in  the  cost  of  V &V. 
Improved  technology  in  these  areas  helps  to 


improve  information  capture  and  flow,  e.g. 
automated  configuration  management  tools. 

8.  Net  Assessment  Of  Issues:  The 
community  needs  to  periodically  reassess  the 
technology  available  for  reducing  the  cost  of 
W&A.  Some  technology  areas  have  a 
potential  for  both  decreasing  and  increasing 
the  cost  of  W&A.  For  example,  advanced 
alternative  representations,  such  as  neural 
nets,  may  reduce  the  size  of  a  model  by 
providing  a  high  level  of  functionality. 
However  they  may  be  more  complicated  to 
imderstand,  making  it  more  difficult  to 
articulate  credibility.  Security  technology 
may  increase  cost  by  limiting  access  to 
information.  Practitioners  need  to  be 
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particularly  careful  about  how  these  will 
impact  W&A  costs.  Cost  benefit  analyses 
need  to  be  done  to  show  that  quality  improves 
over  time. 

It  is  not  clear  how  to  account  for  costs  and  cost 
avoidance  of  W&A  as  compared  with  normal 
M&S  development  costs.  Three  different  cost 
models  are  needed,  on  the  topics  of  legacy,  new 
development  and  distributed  interactive 
simulations.  Cost  factors  need  to  be  standardized 


and  metrics  established  for  W&A  from  standard 
M&S. 

One  possible  source  of  data  for  cost  metrics  is  a 
repository  recently  established  by  the  Navy.  The 
database  has  templates  for  reporting  and 
verification,  vali^tions,  plans  and  reports,  and  is 
hyperlinked.  (For  more  information,  contact  Bob 
Hartling  at  Chief  of  Naval  Operations  (N812D), 
200  Navy  Pentagon,  Washington,  DC  20350- 
2000.) 
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1.  Bacl^ound  In  the  late  1980s,  the  defense 
community  began  a  transition  from  the  mainframe 
era,  in  which  a  few  organizations  developed  and 
used  simulations  in-house,  to  a  workstation  era, 
in  which  those  simulations  were  increasingly 
hosted  on  more  portable  platforms  and  exercised 
by  organizations  that  were  not  involved  in  their 
development.  As  this  transition  occurred,  the 
defense  community  also  came  to  believe  that 
families  of  simulations  could  provide  significant 
cost  savings  for  analysis,  training  and 
engineering.  While  such  capability  promised 
significant  opportunity,  it  also  implied  a 
consistent  separation  of  the  simulation  developer 
from  the  user.  Therefore,  in  order  to  exploit  this 
technology,  the  defense  leadership  understood 
that  it  would  be  necessary  to  establish  procedures 
to  ensure  that: 

•  Users  would  have  access  to  information 
and  methods  that  would  allow  them  to 
assess  the  appropriateness  of  available 
models 

•  Developers  would  collect  and  maintain 
appropriate  information  about  the 
reliability  and  accuracy  of  their  models 
that  would  support  the  user’s  needs 

While  the  software  engineering  literature  of  the 
1980s  provided  standards  for  software 
requirements  development  and  analysis,  it  did  not 
support  two  important  issues:  the  appropriateness 
of  real  world  characterization  and  abstraction  in 
simulations;  and,  support  of  acceptance  testing  on 
a  continuing  basis  (e.g.  by  users  not  involved  in 
initial  development). 


To  address  these  issues,  MORS  sponsored  a 
series  of  activities  on  Simulation  Validation 
(SIMVAL).  The  initial  mini-symposium 
(SIMVAL  I)  in  October  1990  and  subsequent 
workshops,  SIMVAL  n  in  April  1992  and 
SIMVAL  94  in  September  1994,  provided  the 
basis  for  a  broad  set  of  policies,  activities  and 
practices  referred  to  Jis  W&A  of  simulations. 
SIMVAL  99:  Making  W&A  Effective  and 
Affordable  has  provided  an  opportunity  to  assess 
both  technological  advances  and  the  progress  that 
has  been  made  by  the  community  over  the 
intervening  years. 

2.  Conclusion  The  scope  of  the  Synthesis 
Group  included:  technology  and  methodology 
issues  that  are  cross-cutting  in  that  they  have  a 
significant  impact  on  both  V&V ;  and,  issues  that 
impact  on  the  integration  of  W&A. 

It  is  the  general  assessment  of  the  Synthesis 
Group  that,  notwithstanding  substantial  effort  and 
goodwill,  the  community’s  capacity  to  implement 
W&A,  as  envisioned  in  1992,  remains 
disappointing.  Methodologies  remain  immature, 
largely  ad-hoc  and  incomplete.  Supporting 
technologies  are  not  fully  compatible 
methodologically  and  computationally  (hardware 
and  software).  More  alarmingly,  the  collective 
state  of  knowledge  by  practitioners  is 
disappointing.  The  sections  that  follow  discuss 
four  major  issues  that  contribute  to  these 
observations. 
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3.  Introduction.  Efficient  and  effective 
accreditation  decisions  are  sensitive  to  three  key 
elements: 

a.  Large  and  accessible  data  repositories, 
specifically: 

•  Hardware,  software  and  network 
technology 

•  Community  standards  for  documentation 

b.  Automated  reasoning  systems  for  analyzing 
appropriate  data  in  available  repositories 
and  reducing  those  results  to  practical 
(useful)  metrics,  specifically: 

•  Analytical  methods 

•  Software  that  implements  those  methods 
on  automated  databases  /  repositories 

c.  Human  knowledge  (derived  from  training, 
study  and  experience)  about  how  to 
effectively  apply  the  first  two  capabilities 

A  fourth,  and  equally  important  capability  for 
sustaining  an  effective  W&A  program,  is  the 
capacity  to  capture  the  value  of  W&A  activities, 
collectively  and  individually.  The  community’s 
current  inability  to  fully  define  the  implicit 
(hidden)  as  well  as  explicit  costs  and  benefits  of 
individual  W&A  activities  contributes  to  less 
than  effective  investments  and,  to  some  extent,  a 
general  feeling  of  frustration  about  W&A. 

If  DoD  is  to  make  W&A  Effective  and 
Affordable,  it  must  overcome  four  challenges. 
First,  integrating  methodologies  must  be  matured 
to  ensure  that  the  results  of  disparate  V&V 
activities  are  synthesized  into  coherent  metrics 
that  users  can  use  to  understand  the  decision  and 
training  risks  that  are  associated  with  a  given 
model  and  its  intended  use.  While  powerful 
database  technologies  provide  the  essential 


enabling  technology,  this  is  an  analytical  and 
procedural  challenge. 

Secondly,  the  community  needs  to  develop 
sufficiently  expressive  high-level  languages  or 
formalisms  that  will  allow  SMEs,  users  and 
software  engineers  to  communicate  effectively 
using  a  CCM.  This  is  largely  a  technology  issue. 

Third,  the  community  must  develop  consistent 
and  credible  procedures,  derived  from  a  common 
model  of  cost,  for  assessing  the  value  of  W&A 
activities,  especially  hidden  and  deferred  costs. 

Fourth,  the  defense  leadership  must  redouble  its 
efforts  to  institutionalize  good  practice.  This 
challenge  also  applies  to  the  non-defense 
community.  In  both  instances,  investments  in 
infrastructure  (repositories,  networks  and 
training)  are  required. 

4.  Synthesis  Issue  1:  Develop  verification  and 
validation  methodologies  that  S3mthesize  disparate 
analyses  into  coherent  metrics  that  support  the 
accreditation  requirements  of  users. 

As  previously  discussed,  and  illustrated  below, 
the  tinderlying  motivation  for  DoD’s  W^&A 
policies  and  practices  is  to  ensure  that  users  can 
make  well-informed  judgements  about  the 
decision  and  training  risks  associated  with 
simulation-based  activity.  While  the  V&V 
literature  provide  a  substantial  number  of  tools 
and  techniques,  users  continue  to  be  frustrated,  in 
large  part,  because  V&V  reports  are  often 
difficult  to  understand  from  the  user’ s 
perspective.  While  the  implications  are  the  same, 
the  root  causes  are  different  for  V&V. 
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Figure  SG-1  W& A  Perspectives 


As  a  simplified  model,  the  figure  below 
characterizes  how  a  user  conducts  validation 
assessment.  Based  on  a  comparison  of  model 
output  and  observations  drawn  from  the  real 
world,  the  validation  agent  produces  an 


assessment  of  the  model’s  prediction  uncertainty 
(likely  error).  This  value  is  then  compared  to  the 
user’s  tolerance  for  error  (validation  criteria)  to 
make  a  determination  about  the  model’s 
suitability  for  the  intended  application. 


Figure  SG-2  Accreditation  Considerations 


While  the  general  analytical  method  is 
straightforward,  the  science,  mathematics  and 
statistics  communities  have  not  yet  produced  a 
truly  comprehensive  theory  for  quantifying 
prediction  uncertainty  in  simulations.  For  that 
reason,  the  comparison  step  in  model  validation 
remains  largely  qualitative  (face  validation)  and 
ad-hoc.  Because  of  difficulties  with  knowledge 
elicitation  and  conceptual  modeling,  this  practice 
leads  to  further  increases  in  uncertainty  about 
subjective  judgements  as  well  as  increased  cost. 
Although  a  comprehensive  theory  remains 
elusive,  DoD  continues  to  monitor  and  support 
the  relevant  research  to  ensure  that  as  formal 
methods  mature,  they  are  incoiporated  into 
recommended  practice  so  that  Ae  following  user 
questions  can  be  answered. 

•  To  what  extent  does  a  model  preserve 
what  is  known  and  not  known  about  this 
domain? 

•  Was  the  underlying  uncertainty  in  the 
database  preserved? 

•  What  modifications  to  the  model  would 
produce  the  most  signiflcant  impact  on  its 
prediction  uncertainty  (relates  to  the  user’s 
decision  risk  management)? 

•  Are  there  simpler  model  designs  that  have 
relatively  equivalent  levels  of  prediction 
uncertainty  (relates  to  the  user’s  cost 
management)? 

With  respect  to  verification,  while  the  causes  are 
different,  the  implication  is  the  same  —  the  user 
conomunity  is  rarely  provided  a  comprehensive 
and  coherent  metric  that  characterizes  the  overall 
reliability  of  the  computational  model  (hardware 
and  software).  Rather,  verification  reports 
generally  provide  voluminous  information  about 
independent  analyses  on  the  various  software 
development  activities  (requirements,  design, 
software  development  and  implementation 
testing).  While  these  reports  are  often  useful  for 
developers,  they  are  rarely  understandable,  let 
alone  useful,  for  the  user.  There  are  two  general 
reasons  for  this:  first,  comprehensive  methods  do 
not  currently  exist;  and  second,  software  quality 
assurance  standards  are  predominantly  process 
versus  product-based. 

While  the  software  engineering  community  has 
developed  disciplined,  and  in  some  cases  formal, 
methods  for  analyzing  requirements,  designs  and 
software,  these  methods  are  independent  of  each 


other.  Because  they  are  independent,  adequate 
methods  for  analyzing  across  the  development 
process  are  not,  generally,  available.  More 
significantly,  industry  standards  for  Software 
Quality  Assurance  (SQA)  such  as  the  Software 
Engineering  Institute’s  Capability  Maturity  Model 
(SEI CMM)  and  the  International  Standards 
Organization  (ISO)  standards  are  almost 
exclusively  focused  on  assessment  of  the 
developer’s  process.  Consequently,  while  these 
metrics  provide  qualitative  views  of  the 
developer’s  processes,  they  do  not  generate  the 
comprehensive  product-specific  metrics  that  are 
required  by  accreditation  agents.  Rather,  the 
accreditation  agent  is  typically  provided  a 
voluminous  set  of  independent  trouble  reports. 
Given  that  even  professional  software  engineers 
are  rarely  able  to  reduce  such  reports  to  a 
comprehensive  assessment  of  risk,  it  is 
unreasonable  to  expect  model  users  to  do  so. 

In  conclusion,  user  requirements  for  model 
verification  are  different  from  industry 
standards  for  SQA.  While  the  SQA  literature 
provides  many  techniques  that  can  be  used  to 
quantify  software  reliability,  there  is  a  tendency 
for  the  model  development  community  to  equate 
SQA  practice  with  model  verification.  They  are 
two  different  methods.  SQA  is  a  program 
management  activity  that  is  related  to  producing 
the  required  product  on  time  and  within  budget. 
SQA  is  not  an  explicit  W&A  task  and  should  not 
be  budgeted  to  VV&A.  The  software  engineering 
community  in  general,  and  the  defense  M&S 
community  in  particular,  needs  to  keep  this 
difference  in  mind  as  a  basis  for  reassessing 
standards  and  practices  for  verification  reporting 
and  costing. 

5.  Synthesis  Issue  2:  Acquire  languages  and 
technologies  for  developing  consistent  and 
accurate  CMs. 

To  perform  accreditation,  the  user  must 
synthesize  the  validation  agent’s  report  about  the 
fidelity  of  the  model  developer’s  representations 
with  the  verification  agent’s  report  about  the 
reliability  of  the  software/hardware 
implementation.  When  explicit  representation  of 
the  CM  is  not  made,  it  limits  the  extent  to  which 
structured  or  formal  methods  can  be  applied  to 
requirements  analysis  and  testing  across  multiple 
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development  cycles  (see  the  following  figure). 
Because  the  costs  of  fixing  model  development 
problems  increases  non-linearly  over  the  software 


Knowledge  Acquisition 


development  process,  the  cost  implications  of 
incorrect,  inconsistent,  or  ambiguous 
requirements  are  often  substantial. 

Formal  Languages 


Knowledge 
Elicitation /Engineerini 


SG-3  Formal  Lai^ages 


Because  SME  play  an  important  role  in  face 
validation,  implicit  CMs  significantly  influence 
validation  assessments  in  the  current  state  of 
practice.  Inconsistencies  in  the  implicit  mental 
models  used  by  the  SMEs  conducting  validation, 
and  those  generating  requirements,  can  contribute 
to  increased  cost  in  model  development  and 
acceptance  testing.  While  technologies  to  support 
Knowledge  Elicitation  /  Acquisition  (KA)  and 
Knowledge  Engineering  (KE)  have  matured  over 
the  past  20  years,  this  technology  has  not  yet  been 
effectively  applied  to  development  of  suitable 
CMs  for  requirements  development  and 
validation.  It  is  not  clear  whether  that  is  because 
the  current  technology  (case  tools)  remains 
inadequate  or  whether  tools  are  not  being 
adequately  integrated  into  the  state  of  practice. 

The  software  engineering  community  has  also 
struggled  for  years  to  develop  a  sufficiently 
expressive  and  robust  design  language  to  support 
formal  design  analysis  and  test.  The  Unified 
Modeling  Language  (UML)  and  a  broad  range  of 
object-based  design  tools  are  examples  of  this 
technology.  While  the  technology  continues  to 
mature,  and  has  been  generally  useful  for  some 
software  development  projects,  it  has  not  yet 
found  wide  acceptance  for  large-scale,  spatially 
distributed  development  enterprises  and 
environments. 


In  summary,  CMs  play  a  central  role  in 
accreditation  because  they  are  used  to 
characterize  both  the  user’s  understanding  of  the 
battlespace  as  well  as  the  software  engineer’s 
representational  requirements.  Lack  of  a  common 
and  adequately  expressive  CM  almost  always 
leads  to  significantly  increased  model 
development  and  W&A  costs  because  of  an 
inability  to  adequately  manage  model 
requirements  and  their  design  and  testing 
implications.  Although  the  knowledge  and 
software  engineering  communities  have  struggled 
separately  with  the  problem  of  building  CMs  to 
support  requirements  and  softw^e  development, 
these  capabilities  have  not  been  effectively 
applied  to  simulation  development  in  the  defense 
(and  non-defense)  domains.  To  the  extent  that  the 
competitive  nature  of  combat,  as  well  as  the 
dynamic  characteristics  of  military  operations, 
limits  the  utility  of  existing  design  languages,  the 
development  of  a  common,  sufficiently  expressive 
meta-language  and  supporting  technologies  for 
both  domain  experts  and  software  engineers  is 
needed.  We  do  not  believe  such  a  capability 
exists.  Development  of  such  a  capability  could 
significantly  reduce  overall  model  development 
and  W&A  costs  in  DoD. 
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6.  Synthesis  Issue  3:  Identify  and  manage  the 
value  of  W&A  activities 

V&V  activities  allow  the  user  to  understand  the 
risk  of  using  simulation  as  a  decision  or  training 
aid.  As  such,  V&V  activities  should  be 
imdertaken  only  to  the  extent  that  they  help  the 
user  and  his/her  accrediting  agent  quantify, 
understand  and  reduce  that  risk.  The  risk  of  using 
the  simulation  products  inappropriately  can  be 
represented  by  the  equation: 

Risk  =  Probability  of  Failure  *  Impact 

where  probability  of  failure  is  a  measure  of  the 
likelihood  of  the  simulation  behaving  erroneously 
and  impact  is  a  measure  of  the  importance  of  that 
failure,  costs  associated  with  simulation  failures 
and  the  impacts  of  those  failures.  The  total  costs 
of  a  verification  and  validation  program  consist  of 
four  general  components: 

The  costing  of  V&V  of  models  and  simulations 
does  not  now  adequately  reflect  either  the 
incurred  costs  of  a  verification  and  validation 
program,  nor  does  it  address  the  value  of  the 
verification  and  validation  program  in  avoiding 

Total  W&A  Cost  = 

Cost  of  Executing  Validation  + 

Cost  of  Executing  Verification  + 

Cost  of  the  User  of  Being  Wrong  + 

Cost  to  the  Developer  of  '^cleaning  up  the  mess" 

where: 

•  Cost  of  executing  validation  includes  all 
costs  directly  incurred  to  support 
validation  activities 

•  Cost  of  executing  verification  includes  all 
costs  directly  incurred  to  support 
verification  activities  excluding  those 
costs  to  produce  products  in  simulation 
development  that  are  a  normal  part  of 
good  software  engineering  practice 

The  remaining  two  factors  in  the  equation  share 
the  characteristic  that  the  amount  decreases  as  the 
effectiveness  of  the  V&V  program  improves: 


•  Cost  to  the  user  of  being  wrong  includes 
costs  incurred  because  an  incorrect 
decision  (based  on  simulation  results)  was 
made  and  implemented.  It  could  include 
such  costs  as  the  cost  of  correcting 
negative  training,  or  the  costs  of  a  false 
start  on  the  next  phase  of  an  acquisition 
program,  or  the  cost  to  redo  a  simulation 
based  experiment  once  the  failure  is 
discovered  and  corrected 

•  Cost  to  the  developer  of  “cleaning  up  the 
mess”  includes  all  costs  needed  to  change 
the  simulation  software  in  order  to  allow 
the  production  of  better  results  (defect 
correction  costs). 

The  allocation  of  resources  between  V&V 
activities  should  be  based  upon  the  impacts 
identified.  For  example,  if  a  simulation  requires  a 
heavy  investment  in  operator  resources  (salaries, 
travel,  etc.)  then  verification  of  the  mean  times 
between  failures  and  mean  times  to  repair  may 
have  greater  impact  on  success  than  validating  the 
nuances  of  degrees  of  faithfulness  of  the 
simulation  to  the  simulated  system.  On  the  other 
hand,  use  of  simulation  to  support  a  milestone 
decision  in  simulation-based  acquisition  may 
dictate  that  more  attention  be  paid  to  the 
correctness  and  faithfulness  of  simulation  results 
than  to  the  strict  adherence  to  best  software 
engineering  practice. 

An  important,  albeit  often  overlooked,  aspect  of 
model  validation  is  the  reference  data  —  the  real 
world  observations  to  which  the  intermediate  or 
final  outputs  of  the  simulation  are  compared. 
Furthermore,  because  a  model  can  be  no  more 
accurate  than  the  underlying  data  (that  which 
served  to  generate  the  embedded  abstractions 
(algorithms),  investments  in  data  collection  are  a 
critical  component  in  model  fidelity.  Because  of 
their  relatively  high  costs,  however,  live  testing 
and  experimentation  are  often  difficult  to  justify. 
For  that  reason,  it  is  important  to  understand  the 
value  of  data  collection  activities  relative  to 
model  user  requirements  for  fidelity.  As  an 
example,  measurement  records  on  the  flight  path 
of  a  target,  as  well  as  the  trajectory  of  an 
engaging  weapon,  can  only  be  as  precise  as  the 
test  instmmentation  and  data  collection  procedure 
(sampling)  used  in  the  experiment.  Reduced 
fidelity  in  data  collection  contributes  to  variability 
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in  test  data.  Sometimes  this  variance  changes  in 
its  level  of  importance  to  a  user  of  this  data.  In 
summary,  there  is  a  direct  relationship  between 
investimnt  in  data  collection  and  the  costs  of 
being  wrong.  This  tradeoff  analysis  is  an 
important,  albeit  often  overlooked,  aspect  of 
managing  a  credible  W&A  program. 

7.  Synthesis  Issue  4:  Develop  a  mature  and 
robust  state  of  practice 

One  of  the  critical  observations  of  SIMVAL  99  is 
that  the  V&V  community  remains  largely  ill- 
informed  about  the  availability  and  applicability 
of  tools  which  may  support  the  V&V  process. 
This  has  serious  implications  to  the  practice  of 
V&V.  With  respect  to  the  current  state  of 
practice,  we  believe  that: 

•  The  available  methodologies  are  still  not 
well-understood 

•  There  is  only  rudimentary  familiarity  with 
available  tools  and  their  applicability 

•  Comprehensive  sources  of  information  on 
tools  as  well  as  their  applicability  (case 
studies)  are  not  available 

•  There  is  little  available  information  about 
V&V  cost  inq)lications  from  either  the 
developer  or  Ae  user’s  perspective. 


Failure  to  address  Aese  problems  wiU  likely  lead 
to  a  future  wiA: 

•  Lots  of  talk  about  process,  but  little 
progress  for  users 

Limited  support  to  reuse  opportunities  — 
information  m  multiple,  diverse  formats; 

•  A  general  inability  to  implement  efficient 
V&V,  especially  for  large  simulations 

•  Difficulty  in  justifymg  mvestments  in 
V&V 

•  A  continuation,  if  not  mstitutionalization, 
of  Ae  view  Aat  W&A  is  too  costly  and 
too  hard 

In  order  to  effect  change  and  improve  Ae  state  of 
practice,  it  is  imperative  Aat  Ae  community: 

•  Bmld  a  compelling  base  of  knowledge 
through  a  strong  education  curriculum 

•  Improve  access  to  information  —  build 
essential  and  accessible  databases 

•  Build  a  meAodological  framework  Aat 
supports  Ae  effective  application  of 
automated  tools 

•  Publish  case  studies 

•  Develop,  document  and  Asseminate  cost- 
impact  information  to  demonstrate  Ae 
value  of  tools  and  procedures. 
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Conclusion 


The  SMVAL  99  Workshop  tackled  a  difficult 
problem  and  made  good  progress,  as  indicated  by 
the  contents  of  this  report.  This  success  was 
made  possible  by  the  talent  and  dedication  of  its 
participants.  The  challenge  is  now  before  the 
MORS  Sponsors  and  the  SCS  International 
leadership  to  lend  their  support  to  the  SIMVAL 
99  recommendations,  both  the  general  ones 
presented  early  in  the  body  of  this  report  and  the 
more  detailed  ones  contained  in  the  group 
synopses,  so  that  maximum  benefit  may  accrue  to 
the  M&S  communities  inside  and  outside  DoD. 

1.  SIMVAL  99  Sponsorship 
The  initial  collaborative  venture  between  MORS 
and  SCS  International  in  co-sponsoring  SIMVAL 
99  was  facilitated  by  involvement  of  a  number  of 
leaders  from  each  organization.  In  addition  to  a 


significant  number  of  SIMVAL  99  participants 
who  participate  in  both  MORS  activities  and  SCS 
Internationa  conferences,  this  involvement 
included; 

a.  The  SIMVAL  99  Co-chairs  currently  hold 
or  have  held  leadership  roles  in  MORS  and 
SCS  International.  Those  roles  include 
MORS  Director,  Workshop  Chair,  MORS 
Symposium  WG  Chair,  membership  on 
various  MORS  Senior  Advisor  Groups 
(SAGs),  SCS  Director,  Associate  Editor  for 
Validation  of  the  SCS  journal  {Simulation), 
and  General  Chair,  Program  Chair,  and 
Session  Chair  for  SCS  conferences. 

b.  Other  significant  leaders  from  MORS  and 
SCS  International 


From  MORS: 


Natalie  Addison 

Vice  President  (Administration) 

Dennis  Baer 

President 

Alfred  Brandstein 

Director 

Paul  Davis 

Wanner  Laureate 

CAPT  Lawrence  Dick 

Vice  President  (Professional  Affairs) 

CAPT  Robert  Eberth 

Director 

Christine  Fossett,  FS 

Past  President,  Fellow 

Col  Kenneth  Konwin 

Director 

Cynthia  Kee-LaFreniere 

Assistant  Administrator 

Corrina  Ross 

Communications  Manager 

Robert  Sheldon 

President  Elect 

Clayton  Thomas,  FS 

Past  President,  Wanner  Laureate,  Fellow, 
Sponsor’s  Representative  (USAF) 

E.  B.  Vandiver  HI,  FS 

Past  President,  Wanner  Laureate,  Fellow 

Richard  Wiles 

Executive  Vice  President 

From  SCS  International: 


Osman  Balci 

Director  at  Large,  SCS  Representative  to 

Board  of  Winter  Simulation  Conference 

Bruce  Fairchild 

Senior  Vice  President 

William  Waite 

Director  at  Large,  SCS  Representative  to 

MORS 

been  prepared  for  publication  in  PHALANX, 
briefings  based  upon  SIMVAL  99  had  been 
presented  to  the  DMSS  VV&A  Technical  Support 
Team  (the  TST  is  preparing  the  revision  of  the 


2.  SIMVAL  99  Products 

By  the  time  this  report  had  been  completed  (mid- 
February),  an  article  based  upon  SIMVAL  99  had 
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DoD  W&A  RPG)  and  the  DMSO  W&A 
Technical  Working  Group  (TWG),  a  brieflng 
about  SIMVAL  99  prepared  for  the  MORS 
Sponsors,  and  arrangements  made  for  SIMVAL 
99  briefings  at  the  March  1999  Simulation 
Interoperability  Workshop  (SIW),  the  June  1999 


MORS  Symposium,  and  the  July  1999  Summer 
Computer  Simulation  Conference  (SCSC).  In 
addition,  a  plenary  luncheon  talk  on  V&V 
Technology  at  the  1999  SCSC  is  planned  as  is  a 
SIMVAL  99  article  for  Simulation. 
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Appendix  A 

TERMS  OF  REFERENCE  (TOR)* 

MILITARY  OPERATIONS  RESEARCH  SOCIETY  (MORS) 
SIMULATION  VALIDATION  (SIMVAL)  WORKSHOP  99 
Making  W&A  Effective  AND  Affordable 
26-29  JANUARY  1999 

SIMVAL  Workshop  99  is  co-sponsored  by  the 
Society  for  Computer  Simulation  International  (SCSI) 


Background 

Since  the  late  1980s,  MORS  has  led  conceptual  development  for  simulation  validation 
(SIMVAL).  Since  the  last  SIMVAL  workshop  (held  in  1994),  Verification,  Validation,  and 
Accreditation  (VV&A)  have  become  central  themes  in  DoD  modeling  and  simulation.  DoD 
Instruction  5000.61  was  released  in  April  1996  to  set  policy  guidance  for  VV&A.  The  DoD 
VV&A  Recommended  Practices  Guide  (RPG),  published  in  November  1996,  is  a  leading  source 
of  VV&A  guidance  within  DoD  today.  Efforts  are  underway  to  improve  the  RPG,  incorporating 
lessons  learned  from  those  who  have  followed  its  guidance.  Significant  increases  in  formal 
W&A  activity  within  DoD  organizations  have  also  been  noted. 

These  developments  underscore  the  need  for  advancing  the  state-of-the-art  in  VV&A.  Decision 
makers,  program  managers,  simulation  developers,  V&V  practitioners,  VV&A  agents,  V&V 
theorists,  academia,  and  simulation  users  are  all  concerned  with  the  conduct  of  VV&A.  While 
DoD  has  moved  forward  in  the  areas  of  policy  and  procedure,  other  salient  needs  have  yet  to  be 
met.  Most  notably,  the  full  scope  of  current  VV&A  technologies  need  to  be  identified  and 
related  to  the  needs  of  DoD  organizations.  Such  exposure  of  VV&A  technologies  should  include 
not  only  tools  and  methodologies  for  performing  V&V,  but  should  also  address  costs  associated 
with  conducting  W&A  -  and  should  especially  identify  those  technologies  which  can  increase 
V&V  effectiveness  without  increasing  costs  or  decrease  costs  without  degrading  V&V 
effectiveness.  VV&A  technologies  and  methodologies  have  pertinence  for  those  outside  DoD  as 
well  as  for  those  within  the  Defense  community.  MORS  welcomes  the  co-sponsorship  of 
SIMVAL  Workshop  99  by  the  Society  for  Computer  Simulation  International  (SCSI),  an 
organization  that  has  served  simulation  professionals  since  the  1950s,  and  which  will  facilitate 
involvement  in  this  workshop  by  those  outside  the  Defense  community. 

Objectives 

The  SIMVAL  99  Workshop,  Making  W&A  Effective  AND  Affordable,  will  explore  how  to 
ensure  M&S  correctness  and  credibility  in  effective,  efficient,  and  affordable  ways.  The 


*  This  final  version  of  the  TOR  identifies  actual  leaders  of  the  SIMVAL  99  Working  Groups  and  Synthesis  Groups 
(various  circumstances,  such  as  unanticipated  schedule  conflicts,  forced  a  few  changes  from  those  identified  earlier), 
adds  a  few  items  to  the  recommended  reading,  updates  some  of  the  plenary  session  descriptions,  and  deletes  the 
workshop  application  form  and  related  materials. 


Appendix  A-1 


SIMVAL  99  Workshop  is  open  to  both  US  and  non-US  participants.  It  will  examine  current 
technology  supporting  WY&A  and  identify  promising  areas  of  possible  technology  developments 
that  can  enhance  W&A.  Such  enhancements  may  be  achieved  by  making  VV&A  more 
effective  for  a  given  level  of  resources  or  by  allowing  the  same  level  of  W&A  to  be  achieved 
with  reduced  resources.  The  three  specific  areas  that  the  workshop  will  focus  on  are  1) 
verification  technology,  2)  validation  technology  and  methodology,  and  3)  the  impact  of 
technology  on  VV&A  costs.  Although  some  of  the  topics  within  these  three  areas  are  still  being 
developed  within  the  DoD  M&S  community  and  elsewhere  (such  as  the  topic  of  simulation 
fidelity),  the  SIMVAL  99  Workshop  will  maintain  its  focus  on  W&A  technology  and  not  attempt 
to  solve  such  related  issues.  The  SIMVAL  99  Workshop  will  simply  use  concepts  that  currently 
exist  in  such  areas  as  the  Workshop  explores  ways  in  which  technology  can  enhance  or  advance 
W&A  capabilities.  The  emphasis  of  the  SIMVAL  99  Workshop  is  on  VV&A  technology,  but 
this  emphasis  is  in  the  context  of  making  models  and  simulations  more  useful  and  effective  in 
operations  research  and  analysis. 

Agenda 

Day  1  am:  The  agenda  for  this  three  day  unclassified  workshop  will  start  with  a  plenary  keynote 
session  on  the  morning  of  the  first  day.  The  plenary  session,  which  will  be  open  to  a  larger  group 
than  those  who  participate  in  the  full  SIMVAL  99  Workshop,  will  have  four  parts: 

1.  The  Workshop  Chairs  will  welcome  the  workshop  participants  and  provide  a  short 
introduction  and  overview  of  the  SIMVAL  subject  area.  This  overview  will  include  a 
summary  of  issues  and  concerns,  the  workshop’s  objectives,  and  the  agenda. 
SIMVAL  issues  identified  during  previous  MORS  special  meetings,  specifically 
ADSA  96  and  Complexity  in  Modeling  and  Simulation-Linkages,  will  be  addressed 
in  relation  to  the  focus  of  this  workshop. 

2.  CAPT  Dennis  McBride,  Office  of  Naval  Research,  Program  Manager  for  Medical 
S&T  has  agreed  to  be  the  keynote  speaker  for  SIMVAL  99.  The  keynote  address  will 
provide  a  broad  context  for  simulation  validation  and  set  a  series  of  specific 
challenges  before  the  participants. 

3.  A  review  of  V&V  technology  will  be  conducted  to  calibrate  the  participants  about 
current  technology  capabilities.  This  material  will  also  be  included  as  part  of  the 
read-ahead  materials.  Mr.  Robert  Poston,  Aonix,  was  scheduled  to  conduct  the 
review,  but  was  prevented  from  attending  SIMVAL  99  by  illness  in  his  family.  Col 
Kenneth  Konwin,  Director  of  the  Defense  Modeling  and  Simulation  Office  (DMSO), 
served  as  a  surrogate  for  Mr.  Poston  with  a  V&V  technology  review  presentation 
during  the  keynote  plenary  session. 

4.  Dr.  Paul  Davis  of  RAND  will  offer  a  cautionary  perspective  on  VV&A,  noting  issues 
of  research  base,  bureaucratic  mischief  that  impedes  high-quality  analysis,  and 
opposing  trends  in  commercial  off-the-shelf  software,  some  of  which  help  and  some 
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of  which  hurt  the  cause  of  W&A.  He  will  end  with  some  suggested  principles  for 
W&A  related  tools,  illustrating  some  of  the  points  with  examples  from  recent  work. 

Note:  Those  who  participate  in  the  opening  plenary  session  (but  not  in  the  entire  workshop)  will 
be  able  to  interact  with  SIMVAL  99  participants  during  the  afternoon  break  and  mixer  of  Day  1. 
Some  in  this  category  are  expected  to  be  developers  of  V&V  tools  who  want  to  be  available  to 
provide  additional  information  about  their  tools  to  SIMVAL  99  participants.  Space  will  be 
provided  for  small  displays  and  additional  materials  about  V&V  tools  beyond  that  which  is 
provided  in  the  SIMVAL  99  read-ahead  packages  and  workshop  handouts. 

Day  1  pm  and  Dav  2  am:  After  the  plenary  session,  SIMVAL  99  Workshop  participants  will 
break  into  working  groups  for  the  remainder  of  the  first  day  and  for  the  morning  of  the  second 
day  to  focus  on  specific  issue  areas  indicated  below.  A  mixer  will  be  held  on  the  first  day  during 
the  late  afternoon.  Developers  of  V&V  technology  may  be  available  during  the  afternoon  of  the 
first  day  to  provide  additional  information  about  their  technologies.  The  read-ahead  package  will 
also  include  descriptive  information  about  various  V&V  technologies,  including  V&V 
automation  tool  descriptions  in  a  common  format  (more  information  about  this  may  be  found  at 
the  end  of  this  TOR).  Members  of  the  Synthesis  Group  will  participate  in  the  Working  Groups  to 
facilitate  idea  flow  across  the  working  groups. 

Dav  2  pm  &  Dav  3:  Each  working  group  will  outbrief  on  the  afternoon  of  the  second  day, 
allowing  a  draft  synopsis  of  ideas  that  have  been  formulated  across  the  working  groups. 
Working  groups  will  re-form  on  the  morning  of  the  third  day  to  refine  and  modify  their  ideas  in 
light  of  the  draft  synopsis.  Each  working  group  will  provide  a  final  outbrief  on  the  afternoon  of 
the  third  day.  This  session  will  include  a  short  summary  of  the  issues,  concerns,  and 
recommendations  identified  by  the  workshop  participants. 

Dav  4  am:  Working  Group  Co-Chairs  and  Working  Group  Reporters  will  be  part  of  a  Synthesis 
Group  that  will  meet  on  the  morning  of  the  fourth  day  to  finalize  the  draft  of  the  workshop 
report. 

Working  Groups  and  Synthesis  Group 

Working  Group  1:  Verification  Technology,  to  include  automated  requirements/specification¬ 
testing  and  CASE  tools,  as  they  are  used  in  the  context  of  modem  simulation  paradigms  (object 
oriented  based  simulation,  distributed  simulation,  extensive  use  of  AI  and  other  adaptive 
methodologies,  etc.).  A  subset  of  verification  technology  are  descriptive  formats  and  how  such 
interact  (e.g.,  this  is  where  data/meta-data  issues  would  reside).  The  emphasis  of  this  Working 
Group  is  on  the  identification  of  available  tools  and  their  applicability,  especially  to  military  uses 
ofM&S. 

Working  Group  2:  Validation  Technology  and  Methodology  will  focus  on  applications  of 
technology  to  support: 
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f.  Subject  Matter  Expert  (SME)  selection,  orientation,  management,  use,  etc.,  especially  in 
conceptual  validation  processes  (but  also  in  results  validation) 

g.  Correlation  of  representational  fidelity  with  application  needs 

h.  Criteria  for  establishing  needed  levels  of  simulation  fidelity 

i.  Automation/technology  potential  (e.g.,  animation)  to  support  validation  methodology 

j.  Correlation  among  the  various  levels  of  simulation  CMs  (CMMS,  simulation  concept, 
simulation  elements/sub-elements) 

Working  Group  3:  Impact  of  Technology  on  VV&A  Costs,  including  identification  of  VV&A 
cost  elements,  costing  algorithms,  and  rules-of-thumb. 

Synthesis  Group:Provide  a  mechanism  to  ensure  cross-fertilization  of  ideas  among  the 
working  groups,  and  to  integrate  and  synthesize  ideas  from  the  workshop.  Members  of  the 
Synthesis  Group  will  also  participate  in  the  Working  Groups  as  a  means  of  facilitating  this 
conceptual  synthesis  and  integration. 

Products 

In  addition  to  insights  gained  by  SIMVAL  99  participants,  the  primary  product  of  the  workshop 
will  be  a  report  of  the  state-of-the-art  in  simulation  VV&A  technology  and  methodology. 
Current  issues,  concerns,  and  recommendations  will  be  identified.  The  report  will  be  produced 
by  the  following  actions: 

1.  Each  pair  of  Working  Group  Co-Chairs,  in  conjunction  with  their  Working  Group  Reporter, 
will  produce  a  short  summary  document.  This  document  will  include  the  following  items 
and  will  be  submitted  prior  to  departure  from  the  workshop. 

a.  Purpose  of  the  working  group 

b.  Membership  of  the  working  group 

c.  Synopsis  of  the  discussion's  key  points  and  lessons  learned 

d.  Significant  issues,  concerns,  and  recommended  solutions 

e.  Smooth  copies  of  all  visual  materials  (in  both  hard  and  soft  copy) 

2.  Prior  to  departure  from  the  workshop,  the  Synthesis  Group,  in  conjunction  with  the 
SIMVAL  99  Workshop  Chairs,  will  review  and  integrate  the  working  group  reports.  They 
will  draft  a  workshop  sununary  and  an  integrated  list  of  issues,  concerns,  and 
recommendations. 

The  SIMVAL  99  Workshop  Chairs  will  submit  a  report  of  the  workshop  to  the  MORS  office  in 
paper  and  electronic  form  no  later  than  30  days  after  the  workshop  ends.  That  report  will  draw 
upon  the  materials  described  above  and  contain  one  or  more  articles  about  the  workshop  suitable 
for  future  publication  in  PHALANX,  SIMULATION,  and  elsewhere.  The  SIMVAL  99  Workshop 
Chairs  will  also  prepare  a  briefing  package  for  presentation  to  the  MORS  Sponsors  and  at  the 
next  Symposium.  An  appropriate  paper,  based  upon  findings  of  the  workshop,  will  be  prepared 
for  one  or  more  future  SCSI  conferences. 
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Workshop  Organization  Structure 


The  SMVAL  99  Workshop  Co-chairs  will  be  Mrs.  Priscilla  A.  Glasow,  The  MITRE 
Corporation,  and  Dr.  Dale  Pace,  Johns  Hopkins  University  Applied  Physics  Laboratory.  The 
Workshop  Co-chairs  will  control  the  attendance  so  that  it  falls  in  the  range  of  120-150  full 
participants  -  there  will  not  be  a  restriction  on  the  limited  participation.  Active  use  will  be  made 
both  of  members  of  appropriate  MORS  working  groups  and  of  contacts  with  V&V  leaders  from 
outside  MORS,  especially  those  within  the  SCS  International  community,  to  ensure  that 
participants  in  the  Workshop  have  the  requisite  expertise  to  ensure  that  the  Workshop  can 
accomplish  its  objectives. 

Two  Co-Chairs  and  a  Working  Group  Reporter  will  lead  each  Working  Group.  The  Working 
Group  Co-Chairs  will  be  responsible  for  organizing  and  leading  the  working  group  as  well  as  for 
moderating  discussions  and  will  take  responsibility  for  participating  in  the  final  workshop 
synthesis  session  on  the  morning  of  the  fourth  day.  Working  Group  Co-Chairs,  in  coordination 
with  the  SMVAL  99  Co-Chairs,  may  recruit  specific  individuals  to  be  part  of  their  working 
group  in  order  to  ensure  that  requisite  expertise  exists  in  the  group.  Working  Group  Reporters 
are  responsible  for  recording  the  discussion  of  their  respective  working  groups,  noting 
particularly  the  lessons  learned,  issues,  concerns,  and  reconunendations  of  the  participants. 
W^orking  Group  Reporters  are  also  invited  to  the  final  workshop  synthesis  session  on  the  morning 
of  the  fourth  day. 

The  Working  Group  Co-chairs  and  Reporters  are: 

1.  Verification  Technology 

Co-Chairs: 

Robert  O.  Lewis  (Tec-Masters,  Inc.) 

Osman  Balci  (Virginia  Tech) 

Reporter:  Susan  Solick  (TRAC) 

2.  Validation  Technology  and  Methodology 

Co-Chairs: 

Gary  Coe  (IDA) 

Tom  Ruth  (AMS AA) 

Reporter:  Dennis  Laack  (CSC) 

3.  VV&A  Costs 

Co-Chairs: 

Richard  Kuhn  (NIST) 

Bill  Waite  (Aegis  Research) 

Reporter:  William  Jordan  (SMDC) 

Synthesis  Group 

Co-Chairs: 

Simone  Youngblood  (DMSO-JHU/APL) 
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John  Morrison  (LANL) 

Reporter:  Walt  Stanley  (TRW) 

Attendance  at  SIMVAL  99  will  be  by  specific  invitation  from  the  Workshop  Co-Chairs.  Those 
invited  will  be  selected  from  those  who  have  completed  a  request  for  participation  in  the 
workshop  (a  copy  of  the  request  for  participation  form  is  contained  at  the  end  of  the  TOR). 
Priority  for  full  participation  in  SIMVAL  99  will  be  given  to  those  who  (1)  have  current 
experience  and  knowledge  of  verification  and  validation  technologies  and  methods,  (2)  have 
experience  or  insight  related  to  VV&A  costs,  or  (3)  have  near-term  simulation  VV&A  concerns 
and  issues. 

Schedule  and  Fees 

The  workshop  will  be  held  at  the  Kossiakoff  Conference  Center  of  Johns  Hopkins 
University/Applied  Physics  Laboratory  (Laurel,  Maryland)  on  26-29  January  1999  (see  the 
JHU/APL  website  for  directions:  URL  =  http://www.jhuapl.edu/public/visit/locat.htm  ). 

For  full  participation  in  SIMVAL  99,  the  fee  will  be  $180  for  federal  government 
employees  and  $360  for  all  others. 

For  limited  participation  (this  is  participation  in  the  opening  plenary  session  and  the 
Day  1  afternoon  break  and  mixer),  the  fee  will  be  $75  for  federal  government  employees 
and  $  1 50  for  all  others. 

MORS  will  handle  registration  and  financial  arrangements. 

Military  Operations  Research  Society  (MORS) 

101  S.  Whiting  Street,  Suite  202 
Alexandria,  VA  22304-3416 
(703)  751-7290  /  (703)  751-8171  (FAX) 
email:  morsoffice@aol.com 

Suggested  Reading: 

Volume  9  Modeling  and  Simulation,  Technology  for  the  United  States  Navy  and  Marine  Corps 
2000-2035:  Becoming  a  2T’-Century  Force,  report  of  the  Panel  on  Modeling  and  Simulation, 
Committee  on  Technology  for  Future  Naval  Forces,  Naval  Studies  Board,  Commission  on 
Physical  Sciences,  Mathematics,  and  Applications,  National  Research  Council.  Published  by 
National  Academy  Press  (Washington,  DC),  1997. 

Robert  M.  Poston,  Automating  Specification-Based  Software  Testing,  IEEE  Computer  Society 
Press,  1996.  ISBN:  0-8186-7531-4. 

Pertinent  contemporary  W&A-related  papers  and  articles  may  be  found  in  the  proceedings  of 
recent  Simulation  Interoperability  Workshops  (SIWs)  -  these  can  be  downloaded  from  the  SIW 
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website  (URL  =  http://www.sisostds.org/siw/  ),  in  the  proceedings  of  the  summer  and  winter 
conferences  of  the  Society  for  Computer  Simulation  International  (SCSI)  -  check  its  website  for 
more  information  (URL  =  http://www.scs.org  ),  and  in  the  September  1996,  March  1997,  and 
June  1997  issues  of  MORS  PHALANX. 

Some  may  find  the  VV&A  chapter  of  the  recently  published  text.  Applied  Modeling  and 
Simulation:  An  Integrated  Approach  to  Development  and  Operation  (D.  J.  Cloud  and  L.  B. 
Rainey,  eds,  McGraw-Hill,  1998,  ISBN:  0-07-228303-3)  to  be  helpful. 

Orientation  to  formal  methods,  their  costs-benefit,  and  associated  automation  may  be  found  in 
Volume  1  of  NASA-GB-002-95  (Release  1.0),  Formal  Methods  Specification  and  Verification 
Guidebook  for  Software  and  Computer  Systems,  July  1995,  NASA  Office  of  Safety  and  Mission 
Assurance.  This  document  itself  (as  well  as  Volume  2  which  contains  more  detailed 
informaition)  can  be  downloaded  from  a  NASA  website  about  Formal  Methods  website.  Its 
URL  is:  http://eis.jpl.nasa.gov/quality/Formal_Methods/index.html  .  The  more  general  website 
(drop  the  “index.html”)  includes  a  number  of  useful  links  to  other  websites  about  formal 
methods. 


Jim  Van  Buren  and  David  A.  Cook,  “Experiences  in  the  Adoption  of  Requirements  Engineering 
Technologies,”  Crosstalk  (The  Journal  of  Defense  Software  Engineering),  Volume  11  Number 
12  (December  1998),  pp.  3-10.  The  URL  for  Crosstalk  On-Line  is: 

http://www.stsc.hill.af.mil/Crosstalk/crostalk.htnil 


Of  possible  interest  also  is  the  following.  Air  Force  Studies  and  Analyses  Agency  ( AFS  AA) 
sponsored  an  accreditation  support  methodology  for  THUNDER,  with  links  to  DoD  VV&A  and 
AF  VV&A  web  sites.  You  can  access  this  via  AFSAA's  home  page  www.afsaa.hq.af.mil  Path 
from  the  AFSAA's  home  page  is  "Links"  /  "OTHER  AREAS  OF  INTEREST"  /"Verification, 
Validation  and  Accreditation  (VV&A)  Support  for  THUNDER  Campaign  Simulation- 

In  addition  to  its  contents,  other  recommended  reading  materials  will  be  identified  in  the 
SIMVAL  99  read-ahead  materials  and  handout  package.  SIMVAL  99  participants  are 
encouraged  to  review  as  much  of  these  recommended  materials  as  they  can  so  that  the  workshop 
may  make  maximum  progress. 

Vitas  of  SIMVAL  99  Co-chairs  and  Speakers  Scheduled  for  the  Opening  Plenary  Session 

Mrs.  Priscilla  A.  Glasow,  The  MITRE  Corporation,  is  one  of  DoD’s  leading  experts  in  Verification,  Validation  and 
Accreditation  (W&A).  She  is  a  Co-chair  of  the  MORS  Simulation  Validation  (SIMVAL)  workshop  and  a  member 
of  the  MORS  Board  of  Directors.  She  is  also  a  member  of  the  MORS  Modeling  and  Simulation  Senior  Advisory 
Group.  Mrs.  Glasow  began  her  work  in  W&A  while  on  active  duty  and  upon  retirement  from  the  U.S.  Navy, 
became  the  first  W&A  program  manager  for  the  Defense  Modeling  £uid  Simulation  Office  (DMSO).  She 
coordinated  DoD’s  W&A  policy  (DoD  Instruction  5000.61)  and  was  Editor  and  co-author  of  the  DoD  W&A 
Recommended  Practices  Guide.  Mrs.  Glasow  currently  serves  as  the  Lead  Integrating  Agent  fo"” 
EvaluatioiWerification  and  Validation  for  the  Joint  Warfare  System  (JWARS).  She  is  a  member  of  the  DMSO 
W&A  Technical  Support  Team  and  the  W&A  Technical  Working  Group.  She  is  a  member  of  the  Wargame  2000 
rV&V  team  and  supports  a  variety  of  BMDO  programs.  Mrs.  Glasow  is  a  contributing  and  active  member  of 
numerous  professional  organizations  including  the  Institute  for  Operations  Research  and  the  Management  Sciences 
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(INFORMS),  the  International  Test  and  Evaluation  Association  (ITEA),  the  American  Defense  Preparedness 
Association  (ADPA),  and  the  Society  for  Computer  Simulation  (SCS). 

Dr.  Dale  K.  Pace,  a  member  of  the  Principal  Professional  Staff  of  The  Johns  Hopkins  University  Applied  Physics 
Laboratory,  is  a  specialist  in  operations  research,  modeling  and  simulation,  analysis,  and  wargaming.  He  taught  in 
the  Hopkins  graduate  program  in  technical  management  and  at  the  Naval  War  College,  where  he  developed  an 
elective  course  on  technology  and  naval  warfare.  Dr.  Pace  is  co-chair  of  the  Military  Operations  Research  Society 
(MORS)  Simulation  Validation  (SIMVAL)  Workshop,  a  member  of  the  Defense  Modeling  and  Simulation  Office 
(DMSO)  Verification,  Validation,  and  Accreditation  (W&A)  Technical  Working  Group,  one  of  the  original  co- 
chairs  of  the  Distributed  Interactive  Simulation  (DIS)  Workshop’s  W&A  Group,  and  an  initial  members  of  the 
Simulation  Interoperability  Standards  Organization  (SISO)  Simulation  Interoperability  Workshop  (SIW)  Conference 
Committee.  He  leads  the  validation  part  of  the  independent  verification  and  validation  (IV&V)  team  for  Wargame 
20(X)  (WG2K).  He  is  Simulation's  Associate  Editor  for  Validation  and  was  Program  Chair  and  General  Chair  for  the 
Society  for  Computer  Simulation  (SCS)  summer  conferences  in  1991  and  1994  respectively. 

Captain  Dennis  McBridie  is  Research  Area  Manager  for  Biomedical  Science  and  Technology  at  the  Navy  Medical 
Research  and  Development  Commanii.  This  organization  is  currently  transforming  and  is  functioning  as  the  Medical 
Systems  Division  at  the  Office  of  Naval  Research.  Following  graduation  from  Navy  Flight  Surgeon  School  and 
Navy  primary  flight  training.  Captain  McBride  was  designated  a  Naval  Aerospace  Experimental  Psychologist.  He 
has  since  served  at  five  Navy  laboratories,  principally  in  R&D  and  flight  test  of  tactical  aircraft  and  systems,  as 
science  advisor  to  the  Deputy  Assistant  Secretary  of  the  Navy  (C4I),  and  as  Program  Manager  for  modeling  and 
simulation  at  the  Defense  Advanced  Research  Projects  Agency.  Captain  McBride  organized  and  managed  the 
DARPA  programs  in  computer-generated  forces,  intelligent  gateways,  and  synthetic  environments  that  were 
ultimately  demonstrated  as  the  Synthetic  Theatre  of  War  ACTD.  Dr.  McBride  graduated  fi'om  the  Navy  flight  test 
engineering  training  program  and  was  selected  by  the  Navy  as  a  NASA  astronaut  candidate.  His  academic 
background  includes  five  advanced  degrees  including  the  Ph.D.  in  mathematical  learning  theory,  masters*  degrees  in 
systems,  public  administration,  and  experimental  psychology,  and  a  second  Ph.D.  in  the  sociobiology  of  economics 
(1998).  Captain  McBride  has  published  and  presented  more  than  100  scientific  papers  in  the  fields  of  experimental 
and  engineering  psychology,  aeromedicine,  information  technology,  economics  and  political  science,  sociobiology, 
and  flight  test  engineering.  He  is  an  adjunct  professor  at  the  Krasnow  Institute  for  Advanced  Study  where  his 
research  interests  are  concentrated  on  modeling  complex  adaptive  systems.  Military  decorations  include  the  Defense 
Superior  Service,  Meritorious  Service,  and  Joint  Service  Commendation  Medals,  and  he  was  presented  the  L.P. 
Coombes  medal  for  technological  contributions  in  1992. 

Mr.  Robert  M.  Poston  of  Aonix  (New  Jersey  office)  has  invented  and  marketed  software  tools,  participated  on 
software  testing  research  projects,  and  led  national  and  international  standards  efforts.  He  has  written  more  than 
sixty  articles  about  software  quality  and  testing,  and  served  as  a  charter  member  of  the  editorial  board  of  IEEE 
Software,  During  his  career,  he  managed  operations  of  NASA’s  Western  Test  Range  at  Vandenberg  Air  Force  Base 
(CA)  where  he  directed  the  work  of  750  people  preparing  for  missile  launches.  He  headed  software  projects  on  the 
Trident  submarine  while  working  for  RCA.  He  founded  and  presided  for  12  years  over  a  small  company  called 
Progranuning  Environments. 

Dr.  Paul  Davis  has  a  B.S.  from  the  University  of  Michigan  and  a  PhD  in  theoretical  chemical  physics  from  MIT. 
Dr.  Davis  worked  subsequently  at:  the  Institute  for  Defense  Analyses  (IDA)  on  strategic  technology;  at  the  U.S. 
Arms  Control  Agency  on  SALT  II  and  space  issues;  at  the  Office  of  the  Secretary  of  Defense  (Program  Analyses  and 
Evaluation)  as  a  strategic  analyst  and,  as  a  senior  executive,  on  a  combination  of  strategy  and  programs  that  underlay 
creation  of  USCENTCOM  and  the  capabilities  demonstrated  in  Desert  Shield;  and  at  RAND,  where  he  has  worked 
on  strategic  and  defense  planning,  advanced  modeling  and  simulation,  decision  modeling,  and  theories  of  deterrence. 
He  was  editor  and  principal  author  of  a  1994  book  "New  Challenges  for  Defense  Planning,"  that  emphasized 
"capabilities  based  planning"  and  planning  for  adaptiveness.  He  has  written  extensively  on  the  theory  and  practice  of 
multiresolution  (variable-resolution)  modeling.  As  the  architect  of  the  RAND  Strategy  Assessment  System  (RSAS) 
for  global,  theater,  and  operational-level  analytic  war  gaining,  he  has  been  a  major  proponent  and  practitioner  of 
methods  for  planning  under  uncertainty.  Dr.  Davis  wrote  a  widely  circulated  study  for  DMSO  on  VV&A  (reprinted 
in  the  MORS  volume  on  the  subject)  and  has  briefed  extensively  on  methods  and  technologies  to  improve  VV&A. 
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Dr.  Davis  has  served  tours  at  RAND  as  a  program  director  and  corporate  research  manager.  He  currently  devotes 
full  time  to  research,  primarily  on  special  cross-cutting  projects. 
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Appendix  B 
Agenda 


.Tannarv  26***  (Tuesday); 

0730  on:  Registration 

0830-1200:  Plenary  Keynote  Session 

SIMVAL  99  Welcome  (Dr.  Dale  Pace,  SIMVAL  99  Co-Chair)  ,  .  , 

Host  Facility  Welcome  (Dr.  James  Coolahan,  JHU/APL  Asst  Director  for  Modeling  &  Simulation) 
MORS  Welcome  (Mr.  Denny  Baer,  MORS  President) 


SIMVAL  99  Introduction  &  Overview  (Mrs.  Priscilla  Glasow,  SIMVAL  99  Co-Cnair) 

-  summary  of  issues  and  concerns, 

-  workshop  objectives 

-  introduction  of  Working  Group  and  Synthesis  Group  leaders 

Keynote  Address  (CAPT  Dennis  McBride,  Office  of  Naval  Research,  Program  Manager  for  Medical  S&T) 


Review  of  V&V  Technology  (Col  Kenneth  Konwin,  surrogate  for  Mr.  Robert  Poston,  Aonix) 
A  Cautionary  Perspective  (Dr.  Paul  Davis,  RAND) 

SIMVAL  99  Instructions  (Dr.  Dale  Pace,  SIMVAL  99  Co-Chair) 

-  handout  materials 

-  tool  displays 

-  Working  Group  and  Synthesis  Group  rooms  &  schedule 

-  Working  Group  assignments 


1200-1300;  Lunch 

1300-1530:  Individual  Working  Group  Meetings  (Session  A) 

1600-17(K):  General  Session  (verbal  reports  from  Working  Group  leaders)  -  in  Auditorium 

1700-1900:  SIMVAL  99  Mixer 

Note:  Exhibits  of  V&V  technology  and  tools  were  available  for  inspection/discussion  on  Tuesday. 


■Tannarv  27***  (Wednesday); 
0730  on:  Registration 


0830-1200 

1200-1300 

1300-1530 

1530-1600 

1600-1700 


Individual  Working  Group  Meetings  (Session  B)  -  breaks  as  needed 
Lunch 

Individual  Working  Group  Meetings  (Session  C) 

IS  1*6  die 

General  Session  (verbal  reports  from  Working  Group  leaders)  -  in  Auditorium 


■Tannarv  28***  (Thursday); 
0730  on:  Registration 


0830-1200 

1200-1300 

1300-1530 

1400-1430 

1430-1700 


Individual  Working  Group  Meetings  (Session  D)  -  breaks  as  needed 
Lunch 

Individual  Working  Group  Meetings  (Session  E) 

Final  Session  (Reports  from  Working  Group  leaders)  -  in  Auditorium 


January  29***  (Friday);  ,  ■  _ 

0830-1230:  Wrap-up  Session  (Working  Group  Leaders  &  Synthesis  Group  only) 
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Appendix  C 

Participants  and  V&V  Tool  Surveys/Exhibits 

The  appendix  begins  with  the  SIMVAL  99  V&V  tool  surveys  and  identification  of  those  which  were  exhibited  at 
SMVAL  99.  The  appendix  then  identifies  participants  in  SIMVAL  99. 

W&A  Technology  &  Tool  Surveys  and  Related  SIMVAL  99  Exhibits 

There  were  six  responses  to  the  SIMVAL  V&V  tool  survey.  They  are  named  below.  The  two  page  V&V  tool 
survey  response  forms  for  these  tools  are  contained  at  the  end  of  this  appendix.  The  items  below  that  are  listed  in 
bold  italics  had  an  exhibit  table  at  SIMVAL  99. 

•  Accreditation  Support  Site 

•  Data  Verification  Interactive  editor  (DAVIE) 

•  Evaluation  Environment 

•  fWARS  V&V  Database 

•  McCabe  Visual  Testing  Toolset 

•  PerfMETRICS 


He  ^  ^  ^  sic  ^ 

Ms  Natalie  S  Addison 
MORS 

101  S.  Whiting  Street 
Alexandria  VA  22304 
TELEPHONE; 

OFF;  703-751-7290  FAX;  703-751-8171 
Email;  morsvpa@aol.com 

Dr  James  D  Arthur 
Virginia  Tech 

Department  of  Computer  Science 
660  McBryde  Hall 
Blacksburg  VA  24061 
TELEPHONE; 

OFF;  540-231-7538  FAX;  703-231-6075 
Email;  arthur@vt.edu 

MR  Robert  N.  Athay 
NSWCDD 

17320  Dahlgren  Road 
Dahlgren,  VA  22448-5100 
TELEPHONE; 

OFF;  540-653-2773  FAX;  540-653-7999 

MR  Dennis  R.  Baer 
Logicon 

2100  S  Washington  Bivd 
Arlington  VA  22204-5703 
TELEPHONE; 

OFF;  (703)-312-2149  FAX;  {703)-31 2-2780 
Email;  dbaer@logicon.com 


Participants  ********** 

DR  Osman  Balci 
Virginia  Tech 
Computer  Science  Dept 
McBryde  Hall 

Blacksburg  VA  24061 -01 06 
TELEPHONE; 

OFF;  540-231-4841  FAX;  540-231-6075 
Email;  balci@vt.edu 

Mr.  Millard  Barger 
S3I 

STE500 

1700  Diagonal  Road 
Alexandria  VA  2231 4 
TELEPHONE; 

OFF;  703-684-8268  FAX;  703-684-8272 
Email;  millard@s3i.com 

MR  Michael  Borovi/ski 
MITRE  Corporation 
1820  Dolley  Madison  Blvd 
MSW625 

McLean  VA  221 02-3481 
TELEPHONE; 

OFF;  703-883-5216  FAX;  703-883-1370 
Email;  borowski@mitre.org 
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Dirk  Brade 

University  at  Der  Bundeswehr  Munchen 
Institut  fur  Technische  Informatik 
D85577  Neubibert 
49  89-6004-3980 
FAX  4989-6006-2268 

Email:  brade@informatik.unibw-muenchen.de 

DR  Alfred  G  Brandstein 

Marine  Corps  Combat  Development 

Command 

Studies  and  Analysis  Division 
3300  Russell  Road 
QuanticoVA22134 
TELEPHONE: 

OFF:  (703)-784-3235  DSN:  278-3235  FAX:  (703)-784-3547 
Email:  algebra@quantico.usmc.mil 

M.  H.  H.  Brasse 

TNO  Physics  and  Electronics  Laboratory  (TNO-FEL) 

Oude  Waalsderperweg  G3 
250G  JG  The  Hague 
The  Netherlands 
+31-703-740287 
FAX:  +31-703-740652 
Email:  brasse@fel.tno.nl 

Mr  Joseph  D.  Buckley 

Lockheed  Martin  -  Government  Electronics 

Systems 

199  Borton  Landing  Road 
PO  Box  1027 
Moorestown  NJ  08057 
TELEPHONE: 

OFF:  609-722-4630  FAX:  609-273-5700 
Email:  joseph.d.buckley@lmco.com 

Mr  Dorian  Y  Buitrago 
The  Aerospace  Corporation 
PO  Box  92957,  MS/633 
Los  Angeles  CA  90009-2597 
TELEPHONE: 

OFF:  310-336-1132  FAX:  310  336-0536 
Email:  dorian.y. buitrago  @  aero.org 

Mr.  Michael  Campbell 

Project  Performance  Corporation 

Fifth  Floor 

7600  Colshire  Drive 

McLean  VA  22102 

TELEPHONE: 

OFF:  703-395-2198  FAX:  703-395-2200 
Email:  mcampbell@ppr.com 


LTC  Patrick  M  Cannon 
JADS  Joint  Test  Force 
11104  MenaulBlvdNE 
Albuquerque  NM  87112 
TELEPHONE: 

OFF:  (505)-846-1385  DSN:  246-1385  FAX:  (505)-846-0604 
Email:  cannonp@jads.kirtland.af.mil 

MR  Larry  R.  Cantwell 

US  Army  TRACDOC  Analysis  Center 

255  Sedgwick  Ave 

Fort  Leavenworth  KS  66027-2345 

TELEPHONE: 

OFF:  913-684-9224  DSN:  552-9224  FAX:  913-684-9232 
Email:  cantwell@trac.army.mil 

Mr  David  B  Cavitt 
BMH  Associates,  Inc. 

Suite  201 

5424  Robin  Hood  Road 
Norfolk  VA  23513-2441 
TELEPHONE: 

OFF:  757-857-5670  FAX:  757-857-6781 
Email:  cavitl@bmh.com 

Mr.  Joseph  Steve  Chambers 
Space  and  Missile  Defense  Battle  Lab 
Attn:  SMDC-BL-SS 
PO  Box  1500 
Huntsville  AL  35747 
TELEPHONE: 

OFF:  256-955-3580  DSN:  645-3580  FAX:  256-955-5136 
Email:  chamberss@smdc.army.mil 

MR  Peter  H.  Christensen 
The  MITRE  Corporation 
Suite  100 
234  S.  Fraley  Blvd 
Dumfries  VA  22026 
TELEPHONE: 

OFF:  (703)  441-7221  FAX:  (703)  441-1779 
Email:  pchris@mitre.org 

MR  Gary  Q  Coe 
Institute  for  Defense  Analyses 
1801  N  Beauregard  Street 
Alexandria  VA  22311 
TELEPHONE: 

OFF:  410-326-9509  FAX:  703-845-6809 
Email:  gqcoe@chesapeake.net 
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Mr.  Mark  Crooks 
RAND 

1333  H  Street,  NW 
Washington  DC  20005-4707 
TELEPHONE; 

OFF;  202-296-5000  FAX;  202-296-7960 
Email;  Mark_Crooks@rand.org 

DR  Paul  K  Davis 
RAND 

POBOX2138 
1700  Main  St 

Santa  Monica  CA  90407-2138 
TELEPHONE; 

OFF;  (310)-451-6912  FAX;  (310)-451-7066 
Email;  paul.davis@rand.org 

MR  George  DeGovanni 
CACI 

1600  Wilson  Blvd#1300 
Arlington  VA  22209 
TELEPHONE; 

OFF;  (703)-558-0281  FAX;  {703)-875-2904 
Email;  gdegovanni@hq.caci.com 

MR  Michael  B  Dewitz 
Coleman  Research  Corporation 
STE1100 

1700  North  Moore  Street 
Arlington  VA  22209 
TELEPHONE; 

OFF;  (410)-278-6619  FAX;  703-558-7470 
Email;  mdewitz@csc.com 

CAPT  Lawrence  L  Dick 
OPNAV 

Director  of  Navy  Training/Acquisition  Div 
(N75) 

2000  Navy  Pentagon 
Washington  DC  20350-2000 
TELEPHONE; 

OFF;  703-697-0182  DSN;  224-1098  FAX;  703-695-5698 
Email;  Dick.Lee@hq.navy.mil 

Ms  Theresa  Dickerson 
CAS  Inc. 

PO  Box  11190 
Huntsville  AZ  35814-1190 
TELEPHONE; 

OFF;  256-895-7088  FAX;  256-895-7061 
Email;  Theresa.Dickerson  @  cas-inc.com 


CAPT  Robert  WEberthUSNR 

OPNAV  N85/Marine  Corps  Warfighting  Lab 

2006  Hawkins  Ave 

Quantico  VA22134 

TELEPHONE; 

OFF;  703-784-3785  DSN;  278-3785  FAX;  703-784-4421 
Email;  eberthr@mcw1  .quantico.usmc.mil 

Mr  Peter  Eirich 

Johns  Hopkins  University/APL 
Room  13  N  414 
11100  Johns  Hopkins  Road 
Laurel  MD  20723-6099 
TELEPHONE; 

OFF;  240-228-7264  FAX;  240-228-4910 
Email;  peter.eirich@jhuapl.edu 

DR  Bruce  Fairchild 
The  Boeing  Company 
499  Boeing  Blvd 
PO  Box  240002 
Huntsville  AL  35824 
TELEPHONE; 

OFF;  256-461-2743 

Email;  bruce.fairchild@boeing.com 

MR  Joseph  L  Faix 

SAIC/JADS 

STE300 

6200  Uptown  Blvd  NE 
Albuquerque  NM  87110 
TELEPHONE; 

OFF;  505-846-0372  DSN;  246-0372  FAX;  505-846-0603 
Email;  faix@jads.kirtland.af.mil 

MR  Joe  Flores  Jr 
Raytheon  Systems  Company 
1 235  Jefferson  Davis  Hwy 
Arlington  VA  22202 
TELEPHONE; 

OFF;  703413-5337  FAX;  703413-8033 
Email;  jflores@re-ro.com 

Mr  Hirome  Fujio 

Computer  Sciences  Coqjoration 
4815  Bradford  Drive 
Huntsville  AL  35805 
TELEPHONE; 

OFF;  256-876-5316  DSN;  746-5316  FAX;  256-842-0969 
Email;  fujio®  rdbewss.redstone.army.mil 
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DR  Forrest  J  Gibson 
Joint  National  Test  Facility 
(BMDO/JNTE)JNTF/WS 
730  Irwin  Ave 

Falcon  AFB  CO  80912-7300 
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TELEPHONE: 

OFF:  (703)-751-7290  FAX:  (703)-751-8171 
Email:  morsone@aol.com 

MS  Simone  M  Youngblood 

Defense  Modeling  and  Simulation  Office 

(JHU/APL) 

1901  N.  Beauregard  St.,  Suite  500 
Alexandria  VA  22311 
TELEPHONE: 

OFF:  703-824-3436  FAX:  703-998-0667 
Email:  smyoung@msis.dmso.mil 
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MR  Wayne  P  Zandbergen 
SSI 

1700  Diagonal  Road,  Suite  310 
Alexandria  VA  22314 
TELEPHONE: 

OFF:  {703)-684-8268  FAX:  (703)-684-8272 
Email:  wayne@s3i.com 
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SIMVAL  99  M&S  V&V  Tool  Survey  Responses 

The  SIMVAL  99  TOR  version  used  to  announce  the  workshop  requested  standardized  descriptions  of  V&V  tools 
and  provided  a  survey  form  by  which  to  collect  such  tool  descriptions.  The  responses  to  this  M&S  V&V  tool  survey 
are  contained  in  this  part  of  the  SIMVAL  report.  Tools  reported  are  identified  below  and  the  survey  responses  about 
them  follow. 

•  Accreditation  Support  Site 

•  Data  Verification  Interactive  editor  (DAVIE) 

•  Evaluation  Environment 

•  JWARS  V&V  Database 

•  McCabe  Visual  Testing  Toolset 

•  PerfMETRICS 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form 


This  form  is  designed  to  collect  information  about  M&S  V&V  automation  capabilities.  The  information  obtained  by 
responses  to  this  survey  will  be  provided  to  all  participants  in  SIMVAL  99.  Those  who  provide  information  via  this 
form  will  be  provided  an  opportunity  to  have  additional  information  about  their  tools  available  for  SIMVAL  99 
participants  (as  described  in  the  note  at  the  end  of  this  form).  Restrict  response  to  fit  on  form. 


Tool  Name:  Accreditation  Support  Site 

Developer/Provider/POC  [name  &  contact  information  of  individual(s)  and/or  organization]: 
ST  -  Millard  Barger  -  703-684-8268  or  millard@s3i.com 


Brief  description  of  primary  use(s)  for  the  tool  &  what  issues  it  is  intended  to  address: 

Accreditation  support  via  the  internet  supplying  information  about  a  given  model  or 
simulation.  Information  is  organized  to  reflect  and  support  accreditation  requirements. 

Simulation  Phases  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


M&S  Planning  (include  resource  estimation) 

M&S  Requirements 

M&S  Conceptual  Modeling 

M&S  Design 

M&S  Implementation 

M&S  Testing  &  Integration 

_ Unit 

_ Function 

_ Sub-system 

_ System 


_ M&S  Configuration  Management 

_ M&S  Use/Application  &  Maintenance 

X _ M&S  Assessment/Evaluation 

_ M&S  Interoperability/Compatibility 

_ M&S  Modification 

_ V&V  Planning  (include  resource  estimation) 

_x _ V&V  Documentation/Reporting 

_ V&V  Management 

_x _ Accreditation/Certification 

_ Standards  Compliance 

_ Other  (specify): 


Simulation  Environments  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply). 


Simulation  Variety: 

_ Closed  Form 

_ Continuous 

_ Discrete  Event 

_ Real-Time 

_ Human/System/Hardware-in-loop 

_ Distributed  processing 

_ Distributed  simulation 

_ Other  (specify): 


Development  Environment: 

_ Structured 

_ Object-oriented 

_ Formal  system 

_ “Water  Fall” 

_ Evolutionary/Spiral 

_ Rapid  Prototyping 

_ Other  (speciify): 

Software  Language(s)  Which  the  Tool 
Accommodates 


Please  continue  as  appropriate  (use  end  note  for  material  beyond  this  page). 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form  (page  2)  for  (Tool  Name): 
Simulation  Aspects  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


Architecture 

Data 

_ Collection 

_ Reduction 

System/Component  Interfaces 
Human  Interfaces  (e.  g.,  GUIs) 


_ Algorithms 

_ Behaviors 

_ Prototypes 

X _ Management 

_ Test  Planning/Execution 

_ Results  Evaluation 

_ Other  (specify) 
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Tool  Use  Considerations:  What  special  equipment  (host  computer,  operating  system,  network,  configuration, 
etc.)/application  software/facilities/skills  are  required  for  effective  use  of  the  tool? 

PC  internet  access. 


What  training  is  required  (length  and  where  available)  for  personnel  using  the  tool? 


What  is  the  cost  of  the  tool? 


What  M&S  or  kinds  of  M&S  have  used  this  tool  previously? 


Other  Information  About  the  Tool  (references  describing  it,  methods/metrics  employed,  any  special 
relationship  between  this  tool  and  CASE  tools  or  other  software  development/testing  automation,  etc.)  -  use  end  note 
for  material  that  causes  the  disclaimer  to  move  to  the  next  page: 

The  current  implementation  of  this  tool  is  for  the  Air  Force  analytic  campaign  simulation,  THUNDER.  The 
tool  is  accessible  via  the  AFMSRR  as  well  as  the  AFSAA  home  page.  Future  use  is  for  the  Office  of  Aerospace 
Studies  (USAF)  for  the  CFAM  model. 

Disclaimer  of  Endorsement:  Reference  in  SIMVAL  99  materials  to  any  specific  commercial  product,  process,  or 
service  by  trade  name,  trademark,  manufacturer,  or  otherwise,  does  not  constitute  or  imply  its  endorsement, 
recommendation,  or  favoring  by  the  United  States  Government,  by  MORS,  or  by  any  other  organization  or  person. 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form 


This  form  is  designed  to  collect  information  about  M&S  V&V  automation  capabilities.  The  information  obtained  by 
responses  to  this  survey  will  be  provided  to  all  participants  in  SIMVAL  99.  Those  who  provide  information  via  this 
form  will  be  provided  an  opportunity  to  have  additional  information  about  their  tools  available  for  SIMVAL  99 
participants  (as  described  in  the  note  at  the  end  of  this  form).  Restrict  response  to  fit  on  form. 

Tool  Name:  Data  Verification  Interactive  Editor  (DAVIE) 

Developer/Provider/POC  [name  &  contact  information  of  individual(s)  and/or  organization]: 

DMSO  Data  Engineering  Bob  Senko 

1901  N.  Beauregard  St.  S-500  503-324-0607 

Alexandria,  VA  22311  rsenko@msb.dmso.mil 


Brief  description  of  primary  use(s)  for  the  tool  &  what  issues  it  is  intended  to  address: 


PC-based  (Windows)  data  quality  checking  tool.  Checks  accuracy,  completeness, 
etc.  of  data  bases.  Based  on  user-designed  rules.  Operates  on  any  ODBC-compliant  DB. 


Simulation  Phases  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


_ M&S  Planning  (include  resource  estimation) 

_ M&S  Requirements 

_ M&S  Conceptual  Modeling 

_ X _ M&S  Design 

_ M&S  Implementation 

_ X _ M&S  Testing  &  Integration 

_ Unit 

_ Function 

_ Sub-system 

_ System 


_ M&S  Configuration  Management 

.X _ M&S  Use/Application  &  Maintenance 

_ M&S  Assessment/Evaluation 

X _ M&S  Interoperability/Compatibility 

_ M&S  Modification 

_ V&V  Planning  (include  resource  estimation) 

X _ V&V  Documentation/Reporting 

_ V&V  Management 

_ Accreditation/Certification 

X _ Standards  Compliance 

_ Other  (specify): 


Simulation  Environments  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


Simulation  Variety: 

_ X _ Closed  Form 

_ X _ Continuous 

X  Discrete  Event 
_x_  Real-Time 

_ X _ Human/System/Hardware-in-loop 

_ X _ Distributed  processing 

_ X _ Distributed  simulation 

_ X _ Other  (specify): 


Development  Environment: 

_ X _ Structured 

_ X _ Object-oriented 

_ X _ Formal  system 

_x_  “Water  Fall” 

X  Evolutionary/Spiral 
_x_  Rapid  Prototyping 

_ x _ Other  (specify): 

Software  Language(s)  Which  the  Tool 
Accommodates 


Please  continue  as  appropriate  (use  end  note  for  material  beyond  this  page). 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form  (page  2)  for  (Tool  Name): 
Simulation  Aspects  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply) 

_ Architecture  - Algorithms 

pata  _ Behaviors 

_x_  Collection  - Prototypes 

_ Reduction  - -  Management 

_ System/Component  Interfaces  - Test  Planning/Execution 

_ _  Human  Interfaces  (e.  g.,  GUIs)  - Results  Evaluation 

_ Other  (specify 
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Tool  Use  ConsiderdtionS!  What  special  equipment  (host  computer,  operating  system,  network,  configuration, 
etc.)/application  software/facilities/skills  are  required  for  effective  use  of  the  tool? 


Personal  computer,  P-II,  Windows  95/NT,  40  MB  free  disk 
Requires  SQL  anywhere  or  interface  to  DB  language 

What  training  is  required  (length  and  where  available)  for  personnel  using  the  tool? 

N/A.  Excellent  User’s  Manual  and  online  support. 

What  is  the  cost  of  the  tool? 

Free 

What  M(&S  or  kinds  of  M&S  have  used  this  tool  previously? 

New  capability.  Being  integrated  into  CMMS  Toolset.  Requested  by  Trac  WSMR, 
AF/ASCET. 

Other  Information  About  the  Tool  (references  describing  it,  methods/metrics  employed,  any  special 
relationship  between  this  tool  and  CASE  tools  or  other  software  development/testing  automation,  etc.)  -  use  end  note 
for  material  that  causes  the  disclaimer  to  move  to  the  next  page: 

-  operates  on:  Oracle,  IBM  OB-2,  Ingress,  Sybase,  Informix,  and  other  ODBC  compliant  DB  systems. 

-  operates  on:  ACII  fixed  length  files,  other  ASCII  file  formats  such  as  comma-separated  variable 
formations  and  dBase,  FoxPro,  and  Paradox. 

-  User  defines  roles  and  can  either  change  his/her  copy  of  the  data  or  send  a  system-generated  report  to 
the  data  owner  (or  both). 

Disclaimer  of  Endorsement:  Reference  in  SIMVAL  99  materials  to  any  specific  commercial  product,  process,  or 
service  by  trade  name,  trademark,  manufacturer,  or  otherwise,  does  not  constitute  or  imply  its  endorsement, 
recommendation,  or  favoring  by  the  United  States  Government,  by  MORS,  or  by  any  other  organization  or  person. 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form  (To  be  submitted  with  a  Participation  Request  form) 

This  form  is  designed  to  collect  information  about  M&S  V&V  automation  capabilities.  The  information  obtained  by 
responses  to  this  survey  will  be  provided  to  all  participants  in  SIMVAL  99.  Those  who  provide  information  via  this 
form  will  be  provided  an  opportunity  to  have  additional  information  about  their  tools  available  for  SIMVAL  99 
participants  (as  described  in  the  note  at  the  end  of  this  form).  Restrict  response  to  fit  on  form* 

Tool  Name:  Evaluation  Environment™ 


Developer/Provider/POC  [name  &  contact  information  of  individual(s)  and/or  organization]: 


Orca  Computer,  Inc. 

Virginia  Tech  Corporate  Research  Center 
1800  Kraft  Drive,  Suite  111 
Blacksburg,  VA  24060-6370 


Dr.  Osman  Balci,  President 
E-mail:  B  alci  @  OrcaComputer.com 

Tel:  540-961-ORCA(6722) 

Fax:  540-961-4162 

URL:  http://www.OrcaComputer.com 


Brief  description  of  primary  use(s)  for  the  tool  &  what  issues  it  is  intended  to  address: 

Evaluation  Environment™  is  a  general  purpose  software  tool  that  enables  qualitative  and  quantitative 
evaluation  by  using  a  hierarchy  of  indicators;  subject  matter  experts;  analytic  hierarchy  process;  crisp,  fuzzy 
and  nominal  scores;  rule-based  object-oriented  knowledge  specification;  HTML-based  reports;  and 
graphical  representation  of  evaluation  results. 

Evaluation  Environment™  is  intended  to  address  many  issues  including: 

□  Accreditation  and  Quality  Assessment  of  M&S  applications 

□  M&S  Data  Quality  Assessment  and  Certification 

□  Government  Test  and  Evaluation,  and  Certification 

□  Complex  System  Design  Quality  Assessment 

□  Software  Quality  Assessment 

Simulation  Phases  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


M&S  Planning  (include  resource  estimation) 

M&S  Requirements 

M&S  Conceptual  Modeling 

M&S  Design 

M&S  Implementation 

M&S  Testing  &  Integration 

_ x _ Unit 

X  Function 

_ X _ Sub-system 

x  System 


_ M&S  Configuration  Management 

_ M&S  Use/Application  &  Maintenance 

,x _ M&S  Assessment/Evaluation 

_ M&S  Interoperability/Compatibility 

_ M&S  Modification 

_x _ V&V  Planning  (include  resource  estimation) 

,x _ V&V  Documentation/Reporting 

X _ V&V  Management 

_x _ Accreditation/Certification 

_x _ Standards  Compliance 

^x _ Other  (specify):  See  above  list. 


Simulation  Environments  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 

Evaluation  Environment™  is  intended  for  the  quality  assessment  of  any  type  of  modeling  and  simulation. 

Simulation  Variety:  _ Human/System/Hardware-in-loop 

_ Closed  Form  _ Distributed  processing 

_ Continuous  _ Distributed  simulation 

_ Discrete  Event  - Other  (specify): 

_ Real-Time 
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Development  Environment: 

_ Structured 

_ Object-oriented 

_ Formal  system 

_ “Water  Fall” 


_ Evolutionary/Spiral 

_ Rapid  Prototyping 

_ Other  (specify): 

Software  Language(s)  Which  the  Tool  Accommodates 


SIMVAL  99  M&S  V&V  Tool  Survey  Form  (page  2)  for  Tool  Name  (Evaluation  Environment'^^  ) 
Simulation  Aspects  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


Architecture 

Data 

_ Collection 

_ Reduction 

System/Component  Interfaces 
Human  Interfaces  (e.  g.,  GUIs) 


_ Algorithms 

_ Behaviors 

_ Prototypes 

_ Management 

_ X _ Test  Planning/Execution 

_ X _ Results  Evaluation 

_ X _ Other  (specify):  VV&A  and  W&C 
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Tool  Use  Considerations:  What  special  equipment  (host  computer,  operating  system,  network,  configuration, 
etc.yapplication  software/facilities/skills  are  required  for  effective  use  of  the  tool? 

□  Windows  95,  Windows  98,  and  Windows  NT  4.0  operating  systems 

□  State-of-the-art  graphical  user  interface 

□  Easy  to  use 


What  training  is  required  (length  and  where  available)  for  personnel  using  the  tool? 

Minimum  or  no  training  is  required  depending  on  the  user's  computer  skills. 


What  is  the  cost  of  the  tool? 

□  Commercial  Single  User  Licenset  $2,995 

□  Government  Single  User  Licenset  $  1 ,495 

□  University  Single  User  Licenset  S 1 ,495 

□  Site  license  is  negotiable. 

t  Half  price  for  each  additional  user  license. 


What  M&S  or  kinds  of  M&S  have  used  this  tool  previously? 

BMDO  NMD  Project  and  BMDO  V&V  WIPT 

Other  Information  About  the  Tool  (references  describing  it,  methods/metrics  employed,  any  special  relationship 
between  this  tool  and  CASE  tools  or  other  software  development/testing  automation,  etc,): 

A  repository  of  hierarchies  of  indicators  are  under  development  for  areas  such  as  accreditation  of  M&S 
applications,  software  quality  assessment,  and  M&S  V&V  status  characterization.  Contact  the  company  for 
other  hierarchies  and  more  information. 


Table  space  (about  3  ft  x  2  ft)  vAW  be  provided  without  charge  in  the  SIMVAL  99  break  &  mixer  area  for  additional 
descriptive  materials  and/or  display  about  a  tool  -  this  area  can  also  be  manned  by  a  representative  of  the  tool  in  the 
afternoon  of  Day  1  of  SIMVAL  99  (January  26*).  Those  who  avail  themselves  of  this  opportunity  must  be 
registered  for  either  full  or  limited  participation  in  SIMVAL  99.  The  number  who  can  be  accommodated  in  this 
fashion  is  limited  and  will  be  allocated  on  a  first-request  basis  (indicated  on  SIMVAL  99  participation  application). 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form 

This  form  is  designed  to  collect  information  about  M&S  V&V  automation  capabilities.  The  information  obtained  by 
responses  to  this  survey  will  be  provided  to  all  participants  in  SIMVAL  99.  Those  who  provide  information  via  this 
form  will  be  provided  an  opportunity  to  have  additional  information  about  their  tools  available  for  SIMVAL  99 
participants  (as  described  in  the  note  at  the  end  of  this  form).  Restrict  response  to  fit  on  form. 

Tool  Name:  JWARS  V&V  Database 

Developer/Provider/POC  [name  &  contact  information  of  individual(s)  and/or  organization]: 

Jack  Jordan 
BMH  Associates 

5425  Robin  Hood  Road,  Suite  201 
Norfolk,  VA  23513-2441 


Brief  description  of  primary  use(s)  for  the  tool  &  what  issues  it  is  intended  to  address: 

Tool  directly  supports  correlation  of  Derived  Representation  Requirements  (DRR)  and 
Model  Validation  Requirements  (MVR)  to  Unified  Joint  Task  List  measures  and  to  Joint 
Application  Design  (JAD)  packets  and  the  Operational  Requirements  Document  (ORD) 


Simulation  Phases  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


M&S  Planning  (include  resource  estimation) 

M&S  Requirements 

M&S  Conceptual  Modeling 

M&S  Design 

M&S  Implementation 

M&S  Testing  &  Integration 

_ X _ Unit 

_ X _ Function 

_ X _ Sub-system 

_ System 


_ M&S  Configuration  Management 

_ M&S  Use/Application  &  Maintenance 

X _ M&S  Assessment/Evaluation 

_ M&S  Interoperability/Compatibility 

_ M&S  Modification 

_ V&V  Planning  (include  resource  estimation) 

X _ V&V  Documentation/Reporting 

X _ V&V  Management 

X _ Accreditation/Certification 

_ Standards  Compliance 

_ Other  (specify): 


Simulation  Environments  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


Simulation  Variety: 

X  Closed  Form 

_ X _ Continuous 

_ X _ Discrete  Event 

_ X _ Real-Time 

X  Human/System/Hardware-in-loop 

_ X _ Distributed  processing 

X  Distributed  simulation 
_ X _ Other  (specify): 


Development  Environment: 

_ X _ Structured 

_ X _ Object-oriented 

X  Formal  system 
_x_  “Water  Fall” 

_ X _ Evolutionary/Spiral 

_ X _ Rapid  Prototyping 

_ X _ Other  (specify): 

Software  Language(s)  Which  the  Tool 
Accommodates:  Microsoft  Access  97 


Please  continue  as  appropriate  (use  end  note  for  material  beyond  this  page). 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form  (page  2)  for  (Tool  Name): 


Simulation  Aspects  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


Architecture 

Data 

_ Collection 

_ Reduction 

System/Component  Interfaces 
Human  Interfaces  (e.  g.,  GUIs) 


X _ Algorithms 

_ Behaviors 

_ Prototypes 

X _ Management 

_x _ Test  Planning/Execution 

X _ Results  Evaluation 

_ Other  (specify 
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Tool  Use  Considerations:  What  special  equipment  (host  computer,  operating  system,  network,  configuration, 
etc.)/application  software/facilities/skills  are  required  for  effective  use  of  the  tool? 

Microsoft  Access  97,  Internet  connectivity  for  database  replication  and  synchronization.  Also  requires 
Microsoft  Application  Manager  for  Internet  synchronization. 

What  training  is  required  (length  and  where  available)  for  personnel  using  the  tool? 

None  -  very  easy  to  use. 

What  is  the  cost  of  the  tool? 


MS  Access  97  -  $339  MS  Office  Developer  Edition  -  $799. 
What  M&S  or  kinds  of  M&S  have  used  this  tool  previously?  None 


Other  Information  About  the  Tool  (references  describing  it,  methods/metrics  employed,  any  special 
relationship  between  this  tool  and  CASE  tools  or  other  software  development/testing  automation,  etc.)  -  use  end  note 
for  material  that  causes  the  disclaimer  to  move  to  the  next  page: 


Disclaimer  of  Endorsement:  Reference  in  SIMVAL  99  materials  to  any  specific  commercial  product,  process,  or 
service  by  trade  name,  trademark,  manufacturer,  or  otherwise,  does  not  constitute  or  imply  its  endorsement, 
recommendation,  or  favoring  by  the  United  States  Government,  by  MORS,  or  by  any  other  organization  or  person. 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form 


This  form  is  designed  to  collect  information  about  M&S  V&V  automation  capabilities.  The  information  obtained  by 
responses  to  this  survey  will  be  provided  to  all  participants  in  SIMVAL  99.  Those  who  provide  information  via  this 
form  will  be  provided  an  opportunity  to  have  additional  information  about  their  tools  available  for  SIMVAL  99 
participants  (as  described  in  the  note  at  the  end  of  this  form).  Restrict  response  to  fit  on  form. 


Tool  Name;  McCabe  Visual  Testing  Toolset 

Developer/Provider/POC  [name  &  contact  information  of  individual(s)  and/or  organization]: 

Pete  Christensen  -  MITRE 
Gray  Rosse  -  McCabe  &  Assoc. 

Brief  description  of  primary  use(s)  for  the  tool  &  what  issues  it  is  intended  to  address: 


-  Complex  Test  Analysis  -  Verification 

-  Unit  Test  -  Validation 

-  Code  Coverage 

Simulation  Phases  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 

_ M&S  Configuration  Management 

x_  M&S  Use/Application  &  Maintenance 

X _ M&S  Assessment/Evaluation 

_ M&S  Interoperability/Compatibility 

X _ M&S  Modification 

X _ V&V  Planning  (include  resource  estimation) 

X _ V&V  Documentation/Reporting 

X _ V&V  Management 

_ Accreditation/Certification 

_ Standards  Compliance 

_ Other  (specify): 


_ M&S  Planning  (include  resource  estimation) 

_ M&S  Requirements 

_ M&S  Conceptual  Modeling 

_ M&S  Design 

X _ M&S  Implementation  . 

x_  M&S  Testing  &  Integration 

_ Unit 

_ Function 

_ Sub-system 

_ System 


Simulation  Environments  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


Simulation  Variety: 

_ Closed  Form 

_ Continuous 

_  X _ Discrete  Event 

_ X _ Real-Time 

_ X _ Human/System/Hardware-in-loop 

_ Distributed  processing 

_ Distributed  simulation 

_ Other  (specify): 


Development  Environment: 

X  Structured 
X  Object-oriented 

_ X _ Formal  system 

_x_  “Water  Fall” 

_ x _ Evolutionary/Spiral 

_ X _ Rapid  Prototyping 

_ Other  (specify): 

Software  Language(s)  Which  the  Tool 
Accommodates:  C,  C++,  ADA,  JAVA,  Fortran 
VB  COBOL 


Please  continue  as  appropriate  (use  end  note  for  material  beyond  this  page). 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form  (page  2)  for  (Tool  Name): 


Simulation  Aspects  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


Architecture 

Data 

_ Collection 

_ Reduction 

System/Component  Interfaces 
Human  Interfaces  (e.  g.,  GUIs) 


_ Algorithms 

_ Behaviors 

_ Prototypes 

_ Management 

X _ Test  Planning/Execution 

_ Results  Evaluation 

_ Other  (specify 
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Tool  Use  Considerations:  What  special  equipment  (host  computer,  operating  system,  network,  configuration, 
etc.)/application  software/facilities/skills  are  required  for  effective  use  of  the  tool? 

PC  or  Unix  Workstation 

What  training  is  required  (length  and  where  available)  for  personnel  using  the  tool? 

On-site  or  in-plant.  One  day  install  ?  (ramp-up) 

Three-day  structured  test  class 

What  is  the  cost  of  the  tool?  N/A 


What  M&S  or  kinds  of  M&S  have  used  this  tool  previously? 

Navy,  USMC,  Army,  Air  Force,  DoD  /  Command,  Control,  Weapons  Systems 

Other  Information  About  the  Tool  (references  describing  it,  methods/metrics  employed,  any  special 
relationship  between  this  tool  and  CASE  tools  or  other  software  development/testing  automation,  etc.)  -  use  end  note 
for  materid  that  causes  the  disclaimer  to  move  to  the  next  page: 

-  Work  with  GUI  Test  Tools 

-  Work  with  Configuration  Management  Tools 

Disclaimer  of  Endorsement:  Reference  in  SIMVAL  99  materials  to  any  specific  commercial  product,  process,  or 
service  by  trade  name,  trademark,  manufacturer,  or  otherwise,  does  not  constitute  or  imply  its  endorsement, 
recommendation,  or  favoring  by  the  United  States  Government,  by  MORS,  or  by  any  other  organization  or  person. 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form 


This  form  is  designed  to  collect  information  about  M&S  V&V  automation  capabilities.  The  information  obtained  by 
responses  to  this  survey  will  be  provided  to  all  participants  in  SIMVAL  99.  Those  who  provide  information  via  this 
form  will  be  provided  an  opportunity  to  have  additional  information  about  their  tools  available  for  SIMVAL  99 
participants  (sis  described  in  the  note  at  the  end  of  this  form).  Restrict  response  to  fit  on  form. 


Tool  Name:  PerfMETRICS 

Developer/Provider/POC  [name  &  contact  information  of  individual(s)  and/or  organization]: 

David  B.  Cavitt,  Edward  P.  Harvey 
BMH  Associates,  Inc. 

5424  Robin  Hood  Rd.,  Suite  201 
Norfolk,  VA  23513-2441 


Brief  description  of  primary  use(s)  for  the  tool  &  what  issues  it  is  intended  to  address: 

PerfMEDTRICS  is  a  performance  monitoring  application  that  captures  run-time  performance  data  on 
distributed  simulations  (and  others).  It  provides  meaningful  and  relevant  information  for  decision-makers  assessing 
the  impact  of  run-time  performance  on  the  validity  of  simulation  behavior.  Data  collection  relates  to  model, 
simulation  infrastructure,  and  operating  systems  performance. 

Simulation  Phases  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


X _ M&S  Planning  (include  resource  estimation) 

_ M&S  Requirements 

_ M&S  Conceptual  Modeling 

X _ M&S  Design 

X _ M&S  Implementation 

X _ M&S  Testing  &  Integration 

_ Unit 

_ Function 

_ X _ Sub-system 

_ X _ System 


_ M&S  Configuration  Management 

X _ M&S  Use/Application  &  Maintenance 

X _ M&S  Assessment/Evaluation 

_ M&S  Interoperability/Compatibility 

_ M&S  Modification 

X _ V&V  Planning  (include  resource  estimation) 

_ V&V  Documentation/Reporting 

_ V&V  Management 

_ Accreditation/Certification 

_ Standards  Compliance 

_ Other  (specify): 


Simulation  Environments  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


Simulation  Variety: 

_ Closed  Form 

_ Continuous 

_ X _ Discrete  Event 

_ X _ Real-Time 

X  Human/System/Hardware-in-loop 

_ X _ Distributed  processing 

_ X _ Distributed  simulation 

_ Other  (specify): 


Development  Environment: 

_ X _ Structured 

X  Object-oriented 
X  Formal  system 
_x_  “Water  Fall” 

_x_  Evolutionary/Spiral 

_ X _ Rapid  Prototyping 

_ X _ Other  (specify): 

Software  Language(s)  Which  the  Tool 
Accommodates:  Software  Language(s)  which  the 
tool  accommodates:  C,  Motif,  C-I-+ 


Please  continue  as  appropriate  (use  end  note  for  material  beyond  this  page). 
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SIMVAL  99  M&S  V&V  Tool  Survey  Form  (page  2)  for  (Tool  Name): 


Simulation  Aspects  For  Which  the  Tool  Is  Applicable  (please  check  all  which  apply): 


_ Architecture 

_x_Pata 

_ X _ Collection 

_ Reduction 

_ X _ System/Component  Interfaces 

_ Human  Interfaces  (e.  g.,  GUIs) 


_ X _ Algorithms 

_ X _ Behaviors 

_ X _ Prototypes 

_ X _ Management 

_ X _ Test  Planning/Execution 

_ X _ Results  Evaluation 

_ Other  (specify 
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Tool  Use  Considerations:  What  special  equipment  (host  computer,  operating  system,  network,  configuration, 
etc.)/application  software/facilities/skills  are  required  for  effective  use  of  the  tool? 

Currently  runs  on  SGI  and  Linux-based  platforms  Uses  Motif.  Requires  multicast  support.  Requires 
application/simulation  knowledge  in  order  to  insert  instrumentation  code. 

What  training  is  required  (length  and  where  available)  for  personnel  using  the  tool? 

Requires  integration  of  application  specific  performance  information  with  underlying  communications  infrastructure. 
Requires  modification  to  existing  display  interface  (GUI)  to  show  relevant  performance  data. 

What  is  the  cost  of  the  tool? 

What  M&S  or  kinds  of  M&S  have  used  this  tool  previously? 

DARPA  STOW  Tier  3  training  environment.  Air  Force  Distributed  Training  (DMT)  with  virtual  cockpit  simulators. 
Navy  Air  Traffic  Control  training  environment. 


Other  Information  About  the  Tool  (references  describing  it,  methods/metrics  employed,  any  special 
relationship  between  this  tool  and  CASE  tools  or  other  software  development/testing  automation,  etc.)  -  use  end  note 
for  material  that  causes  the  disclaimer  to  move  to  the  next  page: 

[1]  Cavitt,  D.B.,  C.M.  Overstreet,  and  K.J.  Maly.  1997.  A  Performance  Monitoring  Application  for  Distributed 
Interactive  Simulation  (DIS).  In  Proc.  1997  Winter  Simulation  Conference,  Dec.  1997,  421-428.  Assn,  for 
Computing  Machinery,  NY,  NY. 

[2]  Cavitt,  D.B„  J.  Bell,  M.  Checchio,  C.M.  Overstreet,  and  K.J.  Maly.  1998.  Performance  Monitoring  for  the 
Design,  Configurations,  and  Control  of  DIS/HLA  Exercises.  In  Proc.  1998  Spring  Simulation  Interoperability 
Workshop.  98S-SIW-066, 

[3]  Cavitt,  D.B.,  M.A.  Gibson,  E.P.  Harvey,  1998.  HLA/STOW  Interface  Development  to  Support  Distributed  Mission 
Training  (DMT).  In  Proc.  1998  Fall  Simulation  Interoperability  Workshop,  98S-SIW-204. 


Disclaimer  of  Endorsement:  Reference  in  SIMVAL  99  materials  to  any  specific  commercial  product,  process,  or 
service  by  trade  name,  trademark,  manufacturer,  or  otherwise,  does  not  constitute  or  imply  its  endorsement, 
recommendation,  or  favoring  by  the  United  States  Government,  by  MORS,  or  by  any  other  organization  or  person. 
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Appendix  D 

SIMVAL  99  Presentations 

This  appendix  contains  pointers  to  SIMVAL  99  briefings  and  presentations  contained  in  the  SIMVAL  ^  Annex. 
These  consists  of  the  three  presentations  of  the  opening  keynote  plenary  session  on  Tuesday  morning,  briefings  made 
to  SIMVAL  99  working  groups,  and  the  formal  outbriefs  by  the  three  working  groups  and  the  synthesis  ^oup  m  die 
final  plenary  session  on  Thursday  afternoon.  These  unannotated  briefing  materials  supplement  information 
contained  in  the  body  of  the  SIMVAL  99  report. 


D1  —  Plenary  Keynote  Session  Presentations  by  CAPT  McBride,  Col  Konwin,  and  Dr.  Davis 

Additional  biographical  material  may  be  found  about  the  scheduled  speakers  for  the  keynote  session  at  the  end  of  the 
SIMVAL  TOR  in  Appendix  A. 

CAFT  Dennis  McBride  USN  (Program  Officer,  Medical  S&T,  Office  of  Naval  Research)  challenged  SIMVAL  99 
participants  to  think  boldly  about  future  modeling  and  simulation  (M&S)  as  well  as  about  verification  and  validauon 
(V&V)  technology  in  his  keynote  address,  “The  Future  of  Technology  and  W&A  in  Warfighting  Simulation. 
CAPT  McBride  stressed  the  two  vectors  of  computational  capabilities  and  trends  (including  those  from 
bioinformatics)  and  of  warfighting  means  and  methods  as  foci  for  the  W&A  community.  He  emphasiz^  Ae 
problems  associated  with  the  rapid  rates  of  change  in  both  of  these  vectors  and  their  implicauons  for  the  W^ 
immunity  CAPT  McBride  supplemented  his  slides  with  excerpts  from  a  video  program  aired  as  part  of  a  Public 
Broadcasting  System  (PBS)  documentary  on  Karl  Sims'  work  on  Artificial  Evolution. 

Col  Kenneth  Konwin  USAF  (Director,  Defense  Modeling  and  Simulation  Office)  was  a  surrogate  speaker  in  the 
plenary  session.  The  speaker  originally  scheduled,  Mr.  Robert  Poston  of  Aonix,  had  a  family  medical  emergency 
that  precluded  his  attendance.  Col  Konwin’s  briefing  on  “V&V  Technology  Review”  gave  a  high  level  overview  of 
V&V  technology,  and  correlated  those  ideas  with  comments  about  personnel  talent  (a  theme  that  cropped  up  more 
than  expected  in  all  parts  of  the  SIMVAL  99  Workshop),  techniques,  and  tools. 

Dr.  Paul  Davis  (RAND  and  the  RAND  Graduate  School)  offered  a  cautionary  perspective  on  VV&A,  noting  issues 
of  research  base,  bureaucratic  mischief  that  impedes  high-quality  analysis,  and  opposing  trends  in  conunercid  off- 
the-shelf  software,  some  of  which  help  and  some  of  which  hurt  the  cause  of  W&A.  His  presentation,  “Beyond  Box 
Checking:  How  Can  Technolo^  Help  Model  Building  and  W&A?”  suggested  principles  for  W&A  related  tools, 
illustrating  some  of  the  points  with  examples  from  recent  work. 

D2  -  Briefings  to  SIMVAL  99  Working  Groups 

Because  SIMVAL  99  was  workshop  and  not  a  conference,  there  were  relatively  few  briefings  at  it  Briefings  were 
used  in  the  working  groups  to  help  group  participants  to  understand  issues  and  topics  so  that  they  could  better  focus 
their  discussions.  While  the  material  presented  in  this  section  is  substantially  the  same  as  what  was  presented  to  Ae 
working  groups,  there  have  been  a  few  corrections  and  modifications.  In  a  few  cases,  materials  shown  to  a  working 
group  were  not  available  for  this  report. 

Briefings  in  WGl  rVerification  Technology! 

•  D-WGl-l  "M&S  Verification  Technology:  An  Overview"  by  Osman  Balci:  to  identify  the  issues  and 
problems  to  be  addressed  during  the  workshop  (this  material  is  based  upon  the  Bdci  paper  which  was  part 
of  the  SIMVAL  99  Handout  and  which  is  included  in  Appendix  F;  for  that  reason  it  is  not  included) 

•  D-WGl-2  "M&S  Requirements  Verification"  by  Gary  Kollmorgen 

•  D-WGl-3  "M&S  Requirements  Verification"  by  Randy  Michelsen 

•  D-WGl-4  "M&S  Conceptual  Model  Verification"  by  Furman  Haddix 

•  D-WGl-5  "M&S  Design  Verification"  by  Delores  R.  Wallace 

•  D-WGl-6  "M&S  Implementation  Verification"  by  Richard  E.  Nance 
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•  D-WGl-7  "Overview  of  Six  Major  VV&A  Processes  and  Associated  Cost  Models"  by  Robert  0.  Lewis  to  a 
combined  session  for  Working  Groups  1  (Verification  Technology)  and  3  (W&A  Costs) 

Briefings  in  WG2  (Validation  Methodology  &  Technology) 

•  D-WG2-1  “Modular  Semi- Automated  Forces  (ModSAF)  Experimentation  (W&A,  Physical  Models, 
Data)”  by  John  Thomas 

•  D-WG2-2  “A  Construct  for  the  Use  of  SMEs”  by  Scott  Harmon 
Briefings  in  WG3  (Technology  Impact  on  W&A  Costs) 

•  D-WG3-1  “Costing  Of  Model  And  Simulation  (M&S)  Verification,  Validation,  And  Accreditation  (W&A) 
Within  DoD  —  A  Managed  (Marginal)  Investment  Strategy”  by  Robert  Gravitz  and  Bill  Waite 

•  D-WG3-2  (same  as  WGl-7)  "Overview  of  Six  Major  VV&A  Processes  and  Associated  Cost  Models"  by 
Robert  O.  Lewis  to  a  combined  session  for  Working  Groups  1  (Verification  Technology)  and  3  (W&A 
Costs) 

•  D-WG3-2  “Advanced  Distributed  Simulation  (ADS)  Work  Breakdown  Structure  (WBS)  AS  of  13  Jan 
1999”  by  LTC  Cannon 

D3  —  Thursday  Afternoon  Outbriefs 

On  Thursday  afternoon,  leadership  of  the  SIMVAL  Working  Groups  and  Synthesis  Group  briefed  their  findings, 
issues,  and  recommendations  to  SIMVAL  99  participants  in  a  closing  plenary  session.  The  outbriefs  contained  in 
this  report  are  polished  and  expanded  versions  of  the  Thursday  afternoon  outbriefs  and  were  provided  by  the  group 
leaders.  The  body  of  this  report  contains  findings,  issues,  and  recommendations  for  the  SIMVAL  99  Workshop  as  a 
whole.  Conclusions  for  the  Workshop  as  a  whole  are  not  just  a  summation  of  the  findings,  issues,  and 
recommendations  from  the  Working  Groups  and  Synthesis  Group.  In  some  cases,  ideas  from  a  single  working  group 
were  not  seen  as  compelling  by  the  rest  of  the  SIMVAL  99  leadership  and  did  not  become  part  of  the  Workshop’s 
conclusions.  Inclusion  of  the  Thursday  afternoon  outbriefs  in  this  appendix  of  the  SIMVAL  99  report  is  a  method  of 
documenting  the  diversity  of  ideas  addressed  in  the  Workshop. 

•  Working  Group  1:  Two  Outbriefs  (one  by  Dr.  Balci,  the  other  by  Mr.  Lewis) 

•  Working  Group  2 

•  Working  Group  3 

•  Synthesis  Group 

•  SIMVAL  99  Co-Chairs 
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Appendix  E 
Acronyms 


ADS 

Advance  Distributed  Simulation 

AI 

Artificial  Intelligence 

APL 

Applied  Physics  Laboratory  (of  Johns  Hopkins  University) 

A2ATD 

Advanced  Armor  Advanced  Technology  Demonstration 

CASE 

Computer  Aided  Software  Engineering 

CER 

Cost  Estimating  Relationship 

CM 

Conceptual  Model 

CMM 

Capability  Maturity  Model 

CMMS 

Conceptual  Models  Of  The  Mission  Space 

COTS 

Conunercial  Off-The-Shelf 

DARPA 

Defense  Advanced  Research  Projects  Agency 

DB 

Database 

DMSO 

Defense  Modeling  and  Simulation  Office 

DoD 

Department  of  Defense 

DTIC 

Defense  Technical  Information  Center 

FFRDC 

Federally  Funded  Research  and  Development  Center 

GOTS 

Government  Off-The-Shelf 

HITL 

Human  In  The  Loop 

IDE 

Integrated  Development  Environment 

ISO 

International  Standards  Organization 

JADS  JT&E 

Joint  Advanced  Distributed  Simulation  Joint  Test  &  Evaluation 

JADSJTF 

Joint  Advanced  Distributed  Simulation  Joint  Test  Force 

JE 

Joint  Experimentation 

JHU/APL 

Johns  Hopkins  University  Applied  Physics  Laboratory 

KA 

Knowledge  Elucidation  /  Acquisition 

KE 

Knowledge  Engineering 

KDD 

Knowledge  Discovery  In  Databases 

MEL 

Master  Environmental  Library 

ModSAF 

Modular  Semi- Automated  Forces 

MORS 

Military  Operations  Research  Society 

MSIAC 

M&S  Information  Analysis  Center 

MSOSA 

M&S  Operational  Support  Activity 

MSRR 

Modeling  and  Simulation  Resource  Repository 

M&S 

Modeling  and  Simulation 

NSF 

National  Science  Foundation 

ONR 

Office  of  Naval  Research 

OOP 

Object  Oriented  Paradigm 

PM 

Program  Manager 

QA 

Quality  Assurance 

RFP 

Request  For  Proposal 

RPG 

(DMSO)  Recommended  Practices  Guide  (for  W&A) 

R&D 

Research  &  Development 

S&T 

Science  And  Technology 

SBA 

Simulation-Based  Acquisition 

SCSI 

Society  for  Computer  Simulation  International 

SEDRIS 

Synthetic  Environment  Data  Representation  &  Interchange  Specification 

SEE 

Software  Engineering  Environment 

SEI 

Software  Engineering  Institute 

SIMVAL 

Simulation  Validation 
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SME 

Subject  Matter  Experts 

SOW 

Statement  Of  Work 

SQA 

Software  Quality  Assurance 

STOW 

Synthetic  Theater  of  War 

UARC 

University  Affiliated  Research  Center 

UML 

Unified  Modeling  Language 

UUT 

Unit  Under  Test 

V&V 

VerilBcation  And  Validation 

W&A 

Verification,  Validation  and  Accreditation 

4GL 

Fourth  Generation  Language 
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Appendix  F 

Read-Ahead  Package  and  Handout  Items 

This  appendix  contains  the  entire  read-ahead  package  and  an  item  from  the  SIMVAL  99  handout  matenals:  a  pa^r 
by  Osman  Balci,  “Verification,  Validation,  and  Accreditation,”  [Proceedings  of  the  1998  Winter  Simulation 
Conference  (eds.  D.  J.  Medeiros,  E.  F.  Watson,  J.  S.  Carson,  and  M.  S.  Manivannan),  pp.  41-48]  which  has  a  table 
identifying  functional  applicability  of  V&V  techniques. 

Read-Ahead  Package 

Military  Operations  Research  Society  (MORS)  Simulation  Validation  (SIMVAL)  Workshop  99 


Making  W&A  Effective  AND  Affordable 

26-29  JANUARY  1999  at  JHU/APL  Kossiakoff  Conference  Center  (Laurel,  Maryland,  USA) 

SIMVAL  99  is  co-sponsored  by  the  Society  for  Computer  Simulation  International  (SCSI) 

This  read-ahpaH  parkage  is  nrimarilv  identifies  resources  which  should  be  of  special  interest  to  participants ^in 
SIMVAL  99.  All  prosnective  participants  are  encouraged  to  explore  these  resources  prior  to  SDdVAL  99  so  that 
they  can  contribute  most  significantly  to  accomplishment  of  SIMVAL  99  obiectivej. 


As  stated  in  the  SIMVAL  99  Terms  of  Reference  (TOR): 


•  SIMVAL  99  will  explore  how  to  ensure  M&S  correctness  and  credibility  in  effectiye, 
efficient,  and  affordable  ways. 

•  It  will  examine  current  technology  supporting  W&A  and  identify  promising  areas  of  possible 
technology  deyelopments  that  can  enhance  W&A.  Such  enhancements  may  be  achieyed  by 
making  W&A  more  effectiye  for  a  giyen  leyel  of  resources  or  by  allowing  the  same  leyel  of 
W&A  to  be  achieyed  with  reduced  resources. 

•  The  workshop  will  focus  on: 

1)  yerification  technology, 

2)  yalidation  technology  and  methodology,  and 

3)  the  impact  of  technology  on  W&A  costs. 

•  The  emphasis  of  the  SIMVAL  99  Workshop  is  on  W&A  technolo^,  but  this  emphasis  is  in 
the  context  of  making  models  and  simulations  more  useful  and  effectiye  in  operations  research 
and  analysis. 

.  In  addition  to  insights  gained  by  SIMVAL  99  participants,  the  primary  product  of  the 
workshop  will  be  a  report  of  the  state-of-the-art  in  simulation  W&A  technology  and 
methodology.  Current  issues,  concerns,  and  recommendations  will  be  identified. 
Presentations  of  this  information  are  expected  at  the  MORS  1999  symposium  and  at 
appropriate  SCSI  conferences. 

The  materials  in  this  read-ahead  package  are  organized  into  four  section^ 

1)  Computer-Aided  Verification  Tools, 

2)  Tools  for  the  Validation  &  Verification  of  Knowledge-Based  Systems, 

3)  Orientation  to  Formal  Methods,  their  Costs-Benefit,  Associated  Automation,  and 

4)  Suggested  General  Reading  about  W&A. 
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1.  Computer-Aided  Verification  Tools 

The  information  in  this  section  came  from  the  URL  below: 
http://www-cad.eecs.berkeley.edu/~tah/CAV/tools.html 

It  was  prepared  by  Tom  Henzinger  and  last  updated  in  April,  1997,  by  tah@eecs.berkeley.edu^ 

The  following  tools  were  demonstrated  at  CAV  96  or  CAV  97. 

CADP  (Caesar- Aldebaran  Distribution  Package):  a  protocol  validation  and  verification  toolset 

The  Concurrency  Factory:  a  graphical  environment  for  specification,  simulation,  verification,  and  implementation 
of  concurrent  systems 

Fc2Tools  and  Auto/Graph:  a  symbolic/explicit  verification  toolset  for  concurrent  processes 

HyTech:  a  symbolic  model  checker  for  embedded  systems 

Invariant  Checker:  automated  deductive  verification  of  reactive  systems 

Kronos:  formal  verification  of  real-time  systems  based  on  timed  automata  and  temporal  logic 

MDG  (Multiway  Decision  Graphs):  abstract  state  enumeration  for  RTL  functional  verification 

METAFrame:  tool  integration  and  engineering  environment  for  the  organization  and  synthesis  of  large-grain 
software  systems 

MoSeL:  a  decision  procedure  for  monadic  second-order  logic  over  finite  strings 
Mu-cke:  a  model  checker 

Murphi:  finite-state  verification  of  high-level  concurrent  systems,  such  as  protocols,  synchronization  algorithms  and 
memory-model  specifications 

NP-Tools:  general-purpose  verification  toolbox  offering  propositional  and  integer  arithmetic  theorem-proving 
capability 

PARAGON:  a  tool  for  visual  specification  and  verification  of  distributed  real-time  systems 
Partial-order  Package:  an  extension  of  SPIN 

PEP  (Programming  Environment  based  on  Petri  nets):  modeling,  simulation,  analysis,  and  verification  of  parallel 
systems 

PNN  (Product  Net  Machine)  &  SH- Verification  Tool:  a  specification  and  verification  tool  for  cooperating  systems 
PROD:  a  Pr/T-net  reachability  analysis  tool  that  supports  verification  with  partial-order  reductions 
PVS:  a  verification  system 

RTGBL  (Real-Time  Graphical  Interval  Logic)  Tools:  graphical  editor,  satisfiability  checker,  counterexample 
generator,  database  and  proof  manager  for  concurrent  real-time  systems 

SMC:  symmetry-based  model  checker  for  verification  under  fairness  assumptions 
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SPIN:  an  efficient  LTL  model  checker  for  distributed  software  designs 
STeP  (The  Stanford  Temporal  Prover):  verification  of  reactive  and  real-time  systems 
TermiLog:  a  system  for  checking  the  termination  of  queries  to  logic  programs 
Uppaal:  validation  and  verification  tools  for  real-time  systems 

VeriSoft:  automatic  detection  of  coordination  problems  between  concurrent  processes  executing  arbitrary  (e.g.,  C  or 
C-H-)  code 

VIS:  a  system  for  verification  and  synthesis 

Note:  More  verification  tools  can  be  found  at  the  Oxford  Formal  Methods  home  page. 

2.  Took  for  the  Validation  &  Verification  of  Knowledge-Based  Systems 

The  information  in  this  section  came  from  the  URL  below: 
http://www.csd.abdn.ac.uk/~apreece/Research/vvtools.html 

The  focus  is  on  1985-1995  references.  The  information  was  compiled  by  Robert  T.  Plant 
(Department  of  Computer  Information  Systems,  University  of  Miami,Coral  Gables,  FL) 

An  Annotated  Bibliography  on  Validation  &  Verification  of  KBS  is  also  available. 
http://www.csd.abdn.ac.uk/~apreece/Research/vvbiblio.html 

CHECKER 
X.  Yu  &  Biswas 

"CHECKER:  An  Efficient  Algorithm  for  Knowledge-Based  System  Verification” 

In,  Proc.  of  the  Third  International  Conference  on  Engineering  Applications  of  A.I. 

IEA/AIE-90.  1990 

CLINT 

L.  de  RAEDT,  G.  SABLON  &  BRUYNOOGHE 

"Using  Interactive  Concept  Learning  for  Knowledge-Base  Validation  &  Verification  and  test  of  KBS 
Chapter  12, 

pp  177-190,  John  Wiley  &  Sons,  Chichester,  England,  1991. 

COCO 

Stephane  Loiseau 

"A  Method  for  Checking  and  Restoring  the  Consistency  of  Rule  bases" 

Int.  J.  of  Human-Computer  Studies  (1994)  40, 425-442 

CONKRET 

Beatrix  Lopez 

"CONKRET;  A  Control  Knowledge  Refinement  Tool" 

In,  Validation,  Verification  and  Test  of  Knowledge-based  Systems,  pp  191-206 

Edited  by;  Marc  Ayel  &  Jean-Pi^rre  Laurent 

Wiley 
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COVADIS 


M.C.  Rousset 

"On  the  consistency  of  Knowledge-bases:  the  COVADIS  system" 
In  proceedings  of  the  ECA188  Conference  pp  79-84,  Mancher, 
Germany  1988 

COVERATERTTE 

Alun  Preece,  Rajjan  Shinghal,  Aida  Batarekh, 

"Principles  and  Practice  in  Verifying  Rule-Based  Systems", 
Knowledge  Engineering  Review,  Vol.  7,  No.  2, 1992.  ppl  15-141. 

CRSV-CLIPS 

C.Culbert  &  R.Savely 

"Expert  system  verifications  &  Validation" 

Proc.  of  First  AAAI  Workshop  on  V,V  &  Testing.  Palo  Alto,  CA 
August  1988. 

DERIVATION  TOOL 


G.-C.  Roman,  R.F.  Gamble,  and  W.E.,  Ball. 

"Formal  Derivation  of  Rule-Based  Programs", 

IEEE  Transations  on  Software  Engineering,  Vol.  19,  No.  3,  p.  277-296,  March  1993. 

EFG  TRANSLATOR 

L.A.  Becker,  P.G.  Green  &  J.  Bhutinager 

"Evidence  Flow  Graphs  for  V&V  of  Expert  Systems",  NASA  Contractor  Report  181810 
ESC 

Brian  Cragun,  Harold  Steudel, 

"A  Decision  Table  Based  Processor  for  Checking  Completeness  and  Consistency" 

International  Journal  of  Man  Machine  Studies,  No.  26, 1987,  pp633-648. 

EVA 

R. A.  Stachowitz,  C.L.  Chang,  T.S.  Stock,  &  J.B.  Coombs 
"Building  Validation  Tools  for  Knowledge-based  Systems" 

First  Annual  Workshop  on  Space  Operations  Automation  and  Robotics  (SOAR  '87) 

Houston,  Texas.  August  5-7,  1987,  pp209-216 

FEAT 

S. W.  French,  C.  Culbert,  D.  Hamilton 

"Experiences  in  improving  the  state  of  Practice  in  Verification  and  Validation  of  Knowledge-  Based 
Systems" 

Workshop  Notes,  6th  Annual  Workshop  on  V&V:  AAAI  1994,  Washington  D.C.,  pp86-93. 
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FFAAK 


T.V.Cuda&  CP.  Dolan 

"Tool  Aided  Non  Formal  Knowledge  Verification",  Workshop  Notes:  AAAI  Workshop  on  V&V, 
Anaheim.  1991. 

IMVER 

F.  Coener,  T.Berch-Capon,  A.Kent 

"A  Binary  Encoded  Incidence  Matrix  Representation  to  Support  KBS  Verification" 

In:  Working  Notes  7  Workshop  on  V&V  of  KBS  at  AAAI  94 — pp84-93. 

mS-CBR 

T.  Terano  &  K.Kobayshi 

"Changing  the  Traces:  Refining  a  rule-base  by  Genetic  Algorithms" 

UCAI  Workshop  on  V&V  of  Knowledge-based  Systems  Notes  1995,  Montreal,  Canada, 

August  19th  1995. 

KB-REDUCER3 

K.Williamson  &  M.Dahl 

"Knowledge-  based  reduction  for  verifying  Rule  Bases  Containg  Equations" 

In:  Working  Notes:  6  Workshop  on  V.V  of  KBS  at  AAAI,  Washington,  D.C.,  pp  66-71 
A.  Ginsberg  &  L.Rose 

"KB-Reducer:  A  System  that  Checks  for  Inconsistency  and  Redundancy  in  Knowledge-bases", 
Technical  Report,  AT&T  Labs,  1987 

KRFOCL 

Michael  J.  Pazzani 

"A  Set  Covering  Approach  to  Testing  rule-based  expert  systems" 

Workshop  Notes  AAAI  1990 
Boston,  MA 

KRUST 

S:  CRAW 

"Automating  the  refinement  of  knowledge-Based  Systems" 

Ph.  D.  Dissertation,  University  of  Aberdeen,  Scottland.  1991 

KVAT 

O.J.  Mengshoel 

"A  tool  for  incremental  K.Validation  in  a  K.Engineering  Workberch" 

Proc.  of  the  European  Workshop  on  the  verification  &  validation 
of  KBS,  EUROVAN  91.  Cambridge  England,  July  1991  pp  133-146 

MELODIA 

E.  Charles,  &  O.Dubois 

"MELODIA:  Logical  Methods  for  Checking  K-Bases" 

In:  M.Ayel  &  J.P.  Laurant  (Eds):  "Validation  ,  Verification  and  Test  of  Knowledge-Based 
Systems"  Chapter  7,  pp95-105,  Wiley  &  Sons,  Chichester  England  1991 


Appendix  F-5 


MVP-CA 


M.  Mehrotra,  C.Wild 
"Multi-Viewpoint  Clustring  Analysis" 

In:  Working  Notes:  6  Workshop  on  Verification  and  Validation  of  KBS 
at  AAAI  Washington,  D.G.,  pp  52-63 

PATH-HUNTER  and  PATH-TRACER 

A.D.Precce,  C.Grossner,  P.G.Chander,  T.Radhakrishnan 
"Structural  validation  of  Expert  systems  using  a  formal  Model” 

In:  Working  Notes:  Workshop  on  V&V  of  KBS  at 
AAAI  93,  Washington  D.C.  August  1993,  pp  19-26 

PREPARE 


D.  Zhang  &  D.  Nguyen 

"PREPARE:  A  Tool  for  Knowledge-base  Verification" 

IEEE  Transactions  on  Knowledge  and  Data  Engineering  ,  Vol.  6,  No.  6.  December  1994,  pp983-989 
PROLOGA 
J.  Vanthienen 

"Knowledge  Acquisition  and  Validation  Using  a  Decision  Table  Engineering  Workbench" 

World  Congress  of  Expert  Systems  1991,  ppl  861-1 868 

RTTCaG 

U.C.  Gupta  &  J.BIEGEL 

"RTTCaG:  A  Rule-Based  Intelligent  Test  Case  Generator" 

In,  Working  Notes,  AAAI  Workshop  on  ICBS,  Verification, 

Validation  and  Testing:  July  1990,  Boston  MA. 

SACCO/SYCOJET 

Marc  Ayel  &  Jean  Pierre  Laurent 

"SACCO-SYCOJET:  Two  Different  Ways  of  Verifying  Knowledge-Based  Systems" 

In,  Validation,  Verification  and  Test  of  I^owledge-based  Systems,  pp  62-76 

Edited  by:  Marc  Ayel  &  Jean-Pierre  Laurent 

Wiley 

SAVES 

S. Smith  &  A.Kandel 

Verification  and  Validation  of  Rule-based  Expert  Systems 
CRC  Press 

SOCRATES 

B  Traylor,  U.Schwuttke,  A.Quan 

"A  Tool  for  American  verification  of  real-Time  Expert  systems" 

In:  Working  Notes  7  Workshop  on  V&V  of  KBS  at  AAAI  94, 

Seattle,  WA,  August  1994,  pp  79-83 
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SPT 

R.  Phelps,  W.  Aerts 

“Improving  Validation  and  Verification  of  Knowledge-Based  Systems  Through  Naturally 
Comprehensible  Flow  representations" 

In:  EVROVAV  '93,  Proceedings  of  the  European  Symposium  on  the  Validation  and  Verification  of 
Knowledge-Based  Systems,  Eds:  J.  Cardenosa,  &  P.  Meseguer,  pp  295-309 

SYSIFE 

P.  Mazas 

"Designing  Knowledge  Validation  Through  Experimentation:  The  SYSIFE  System" 

In:  M.  Ayel  &  J.P.  Laurent  (Editors),  Validation,  Verification  and  Test  of  Knowledge-based 
Systems,  pp  1 19-145.  Wiley  1991 

TRUBAC 

"Rule-Based  Coverage  Measures  Applied  to  Testing  Rule-Bases  with  Uncertainty" 

IJCAI  Workshop  on  V&V  of  Knowledge-based  Systems  Notes  1995,  Montreal,  Canada, 

August  19th  1995. 

VALID 

J.  Cardevosa,  N.  Juisto 

"General  Overview  of  the  VALID  Project" 

In:  EVROVAV  ’93,  Proceedings  of  the  European  Symposium  on  the  Validation  and  Verification  of 
Knowledge-Based  Systems,  Eds:  J.  Cardenosa,  &  P.  Meseguer,  pp53-67 

VALIDATOR 

Y.Kang  &  T.Bahill 

"A  Tool  for  Detecting  Expert  System  Errors" 

AI  Expert,  Feb.  1990  pp42-51 

VASTT 

M. Steib,  R.Small,  C.Castells-Schofield. 

"Tailoring  VASTT  for  Expert  system  verification,  validation  and  testing" 

Workshop  Notes,  AAAI  Workshop  on  V  &  V  ,  1991. 

VITAL 

Alain  Rouge,  Jean  Yves  Lapicque,  Florent  Brossier,  Yves  Lozinguez 
"Validation  and  verification  of  KADS  Data  &  Domain  Knowledge" 

EUROVAV  ’93,  pp69-83 

Palma  De  Mallorca,  Spain  24-26th  March  1993 

Univ.  Polit.  de  Madrid  (UPM) 

WR 

N.  Zlatarova 

"WR:  A  Uniform  Framework  for  Expert  System  Knowledge  Bases  Verification,  Validation  and 
Refinement" 

Workshop  Notes:  AAAI  Workshop  on  V&V,  Anaheim  1991. 
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WRAPPINGS 


K.L.  Bellman  &  C.  Landauer 

"The  Modeling  Issues  Inherent  in  testing  and  Evaluating  K-Based  Systems" 

Expert  systems  with  Applications  Journal  Vol,l,  pp  199-215  (1990). 

3.  Orientation  to  Formal  Methods,  their  Costs-Benefit,  Associated  Automation 

Orientation  to  formal  methods,  their  costs-benefit,  and  associated  automation  may  be  found  in  Volume  1  of  NASA- 
GB-002-95  (Release  1.0),  Formal  Methods  Specification  and  Verification  Guidebook  for  Software  and  Computer 
Systems,  July  1995,  NASA  Office  of  Safety  and  Mission  Assurance.  This  document  itself  (as  well  as  its  associated 
Volume  2  which  contains  more  detailed  information  about  formal  methods)  can  be  downloaded  from  a  NASA 
website  about  Formal  Methods.  The  URL  of  this  website  is: 

http://eis.jpl.nasa.gov/quality/Formal_Methods/index.html  .  The  more  general  website  (drop  the  “index.html”) 
includes  a  number  of  useful  links  to  other  websites  about  formal  methods. 

4.  Suggested  General  Reading  about  VV& A 

Volume  9  Modeling  and  Simulation,  Technology  for  the  United  States  Navy  and  Marine  Corps  2000-2035: 

Becoming  a  2F'-Century  Force,  report  of  the  Panel  on  Modeling  and  Simulation,  Committee  on  Technology  for 
Future  Naval  Forces,  Naval  Studies  Board,  Commission  on  Physical  Sciences,  Mathematics,  and  Applications, 
National  Research  Council.  Published  by  National  Academy  Press  (Washington,  DC),  1997. 

Robert  M.  Poston,  Automating  Specification-Based  Software  Testing,  IEEE  Computer  Society  Press  1996  ISBN- 
0-8 186-753 1-4. 

Pertinent  contemporary  W&A-related  papers  and  articles  may  be  found  in  the  proceedings  of  recent  Simulation 
Interoperability  Workshops  (SIWs)  -  these  can  be  downloaded  from  the  SIW  website  (URL  = 
http://www.sisostds.org/siw/ ),  in  the  proceedings  of  the  summer  and  winter  conferences  of  the  Society  for  Computer 
Simulation  International  (SCSI)  ~  check  its  website  for  more  information  (URL  =  http://www.scs.org  ),  and  in  the 
September  1996,  March  1997,  and  June  1997  issues  of  MORS  PHALANX. 

Some  may  find  the  W&A  chapter  of  the  recently  published  text.  Applied  Modeling  and  Simulation:  An  Integrated 
Approach  to  Development  and  Operation  (D.  J.  Cloud  &  L.  B.  Rainey,  eds,  McGraw-Hill,  1998,  ISBN:  0-07- 
228303-3)  to  be  helpful. 

Patrick  J.  Roache,  Verification  and  Validation  in  Computational  Science  and  Engineering,  1998  (ISBN  0-913478- 
08-3).  Available  via: 

http://kumo.swcp.com/hermosa/html/books.html 

This  book  covers  modem  approaches  to  Verification  and  Validation  of  computer  codes  and  other  topics  related  to 
the  Quantification  of  Uncertainty  and  code  Quality  Assurance.  Especially  noteworthy  are  the  Method  of 
Manufactured  Solutions,  a  general  method  for  obtaining  exact  solutions  for  code  Verifications,  and  the  Grid 
Convergence  Index,  a  method  for  uniform  reporting  of  grid  convergence  tests.  Applicable  to  commercial  codes  and 
to  all  problems  modeled  by  partial  differential  equations,  examples  include  CFD,  Groundwater  Modeling,  etc. 
Provides  useable  formulas  and  practical  guidance  for  error  estimation  and  experimental  Validation. 

Of  possible  interest  also:  William  Hankley’s  Software  Quality:  Assurance,  Validation,  and  Verification,  1998  (see: 
http://www.cis.ksu.edu/~hankley 

His  notes  on  vocabulary  are  interesting: 

Validation  and  verification  (V&V)  are  two  specific  aspects  of  the  broader  area  of  software  quality  assurance  (SQA). 

There  are  different  working  definitions  for  V&V. 

Boehm  [81]:  Validation  ...  does  the  product  have  the  right  functions  and  features; 
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Verification  do  the  functions  work  correctly. 

lEEE/ANSI:  Validation  ...  the  process  of  evaluating  a  system  or  component 

during  or  at  the  end  of  the  development  process 
to  determine  whether  it  satisfies  specified  requirements. 

Verfication.  ..the  process  of  evaluating  a  system  or  component 
to  determine  whether  the  products  of  a  given 
development  phase  satisfy  the  conditions  imposed  at  the 
start  of  that  phase. 

E.  Kit:  Validation  ..  involves  executing  the  actual  software  or  a  simulated 

mock-up; 

it  is  a  "computer-based  testing"  process; 
it  usually  exposes  symptoms  of  errors. 

Verification.,  involves  evaluating,  reviewing,  inspecting,  and  doing 
desk  checks  of  work  products  such  as  requirements, 
designs,  and  code  (static  analysis  —  not  execution) ; 
it  is  a  form  of  "human"  testing. 

84 1 :  Validation  ..  any  well  defined  method  for  checking  and  or  enhancing 

attributes  of  quality  of  software 

Verification,,  any  well  defined  method  for  demonstrating  or  assuring 
the  agreement  of  descriptions  of  software  attributes. 

Note  that  these  definitions  are  not  consistent 

Boehm’s  definitions  do  not  address  issues  of  "quality"  that  only  indirectly  relate  to  software  functions. 

For  the  TFFF  definitions,  aspects  of  "quality”  can  be  included  as  requirements. 

Kit's  definitions  are  not  appropriate;  V  &  V  can  both  be  done  manually  or  using  computer  tools. 

By  the  841  definition: 

Validation  includes  testing  of  software. 

Since  testing  is  usually  not  exhaustive,  it  is  not  considered  to  be  a  verification  of  software. 

Validation  depends  upon  definitions  and  measures  of  quality  of  software. 

Validation  also  includes  aspects  of  SQA.  such  as  software  process  plans,  reviews,  and  tools. 

Verification  covers  various  forms  of  "proof  of  correctness"; 

such  "proof’  may  be  formal  (using  mathematical  models)  or  informal  (using  inspections  and  reviews). 

The  model-checking  method  of  verification  uses  exhaustive  testing  of  a  model  to  prove  that  some 

property  holds. 

Some  verification  may  be  "weaker"  than  some  validation  in  that  the  verification  may  only  prove  one 

small  property. 

SIMVAL  99  Handout  Item:  a  paper  by  Osman  Balci,  ‘‘Verification,  Validation,  and  Accreditation” 
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